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FOREWORD 


The Design of Training Systems (DOTS) project objectives are in consonance 
with the requirements of Advanced Development Objective ZPN07 (formerly 
ADO 43-03X), Education and Training Developmen. ZPN07 includes a number 
of projects concerned with demonstrating and evaluating the technical, opera¬ 
tional and financial feasibility of applying advanced technological applications 
to improving the training process. 

The Bureau of Naval Personnel initiated the original ADO in 1966 to make Naval 
training more responsive to the changing times. As one project under this effort, 
DOTS was designed to improve the process of managing training resources through 
application of the techniques of system analysis and system simulation as accom¬ 
plished through mathematical modeling. The end objective is a family of computer¬ 
ized mathematical models enabling training management to more rapidly predict 
the impact of changes in training resource availability or requirements. 

The majority of education and training was reorganized in 1971 under one com¬ 
mand, Chief of Naval Education and Training (CNET). Because of this change, 

DOTS responsibility was transferred to CNET in March of 1972, more specifically 
to the Training Analysis and Evaluation Group (TAEG), Orlando, Florida. The 
new CNET organization greatly increased the potential benefits to be gained 
from the increased application of new management techniques and, therefore, 
from the DOTS' R&D effort. Accomplishment of DOTS began in February of 1973 
with the majority of tasking being assigned to the International Business 
Machines Corporation, Federal Systems Division, Cape Kennedy Facility, located 
at Cape Canaveral, Florida. 

In conducting the Phase I study and definition effort, the TAEG/IBM technical 
team conducted multi-level interviews at some eighty activities or training 
related groups within the Naval Education and Training Command (NAVEDTRACOM). 

The willing and competent participation of all personnel contacted is gratefully 
acknowledged. Special recognition is due the Fleet Training Center, Norfolk, 
Virginia, which was selected as a test bed for the DOTS' models. All levels of 
the command were exceptionally cooperative and willing to contribute significant 
time and interest to the implementation of the models and their subsequent test 
and evaluation during Phase III. 

The SCRR, ETE, and TPF models were developed by Mr. R. Yanko, Mr. H. Bellamy, 
and Mr. K. Branch, respectively. Systems Programming support was provided to the 
modelers by Mr. J. Chapman and Mr. J. Staley. Mr. C. Edison developed and im¬ 
plemented the Phase III training program. All team members participated in the 
Phase III documentation effort, with Mrs. D. Gardner, Mrs. L. Girard, Mrs. C. 
Reilly and Mrs. E. Taylor providing editorial and secretarial services. 

Mr. R. Hallman was Project Manager. 

The Training Analysis and Evaluation Group, Dr. A. Smode, Director, project 
team members Mr. H. Okraski, Mr. T. McNaney and Mr. W. Lindahl, complemented 
the contracted effort by providing guidance and establishing organizational 
interfaces. 
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SECTION I 
INTRODUCTION 


BACKGROUND 

The Department of Defense (DoD) Is faced with maintaining a strong national 
defense posture despite declining allocations and the impact of world 
inflation. The increasing cost of complex weapons systems and support man¬ 
power is significantly increasing the challenge of meeting national defense 
objectives. 

Approximately 14% 1 of the DoD annual budget is allocated to some form of 
education or training. This represents about 25 % of DoD's total manpower 
budget. Obviously, a major strategic thrust of the military services is to 
reduce education and training costs to a minimum, while maintaining the re¬ 
quired level of effectiveness. 

The majority of Naval training is under the command of the Chief of Naval 
Education and Training (CNET). The CNET is responsible for providing the 
training support required to maintain fleet readiness within the constraints 
of available funding. The Design of Training Systems (DOTS) project Is intended 
to assist the Naval Education and Training Command (NAVEDTRACOM) In achieving 
this goal. 

The overall DOTS' objective Is to provide NAVEDTRACOM management with additional 
tools in the form of computerized mathematical models to asssist in predicting 
the quantitative impact of training resource decisions. The planning process 
will be enhanced by providing decision makers with the capability to economically 
and rapidly consider a wider range of alternatives. The project stresses a 
step-wise progression from system analysis, through development of computer- 
based test models of selected sub-elements within the NAVEDTRACOM, to recom¬ 
mendations for making the models operational. 

Phase I of the DOTS project was completed in December 1973. It Included: 

(1) a comprehensive functional analysis of the NAVEDTRACOM, (2) development 
of a set of strategic assumptions describing the environmental elements expected 
to affect the NAVEDTRACOM In the 1980's, (3; development of recommendations 
leading to an idealized training system in terms of the projected needs of the 
1980's, and (4) creation of a candidate list of potential computer-based math¬ 
ematical models to permit selection of a maximum range test configuration. If 
further information is desired, consult the DOTS Phase I Pinal Report 2 . 

Phase II was completed In October 1974. It Included: (1) selection of three 
test models, from the candidate list developed during Phase I, representing 


^ Defense Space Daily , 17 January 1974, Page 92. 

^ Design of Training Systems, Phase I Final Report , TAEG Report No. 12-1, 
December 1973. 
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a range of subsets of the NAVEDTRACOM resource planning and management process, 
(2) development and logical validation of the selected test models, (3) iden¬ 
tification of those areas within the NAVEDTRACOM which could not be modeled 
nor well defined, suitable modeling techniques do not exist, or data for model 
input or testing is not readily available, (4) development of a hypothetical 
cost avoidance analysis projecting the impact of modeling tools operating 
within a modified NAVEDTRACOM management system, (5) development of the 
design for verification studies to be carried out during Phase III. The 
three selected test models were the System Capabilities/Requirements and 
Resources (SCRR), the Educational Technology Evaluation (ETE), and the Train¬ 
ing Process Flow (TPF) models. Additional information may be obtained from 
the DOTS Phase II Final Report 3 . 

Phase III was completed in October 1975. Phase III included development of the 
unique software required to implement the DOTS' models in the Norfolk test bed, 
the orientation and training of operating personnel and members of the Govern¬ 
ment Test and Evaluation (T&E) team, support of the T&E, and final user and 
maintenance documentation. 

The majority of the technical data concerning the three DOTS models is contained 
in the previously mentioned documentation. A brief description is presented in 
SECTION IV, PHASE III PRODUCTS. The emphasis of this Phase III report will be 
on describing the T&E task, summarizing its results and suggesting recommenda¬ 
tions for future DOTS direction. 


3 - 

Design of Training Systems, Phase II Final Report . TAEG Report No. 12-2, 
December 1974. 
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PROBLEM STATEMENT 

The direct result of the DOTS effort, as implied by its objective, is envisioned 
as a family of computerized mathematical models providing a new capability to 
predict the quantitative impact of training resource decisions. The indirect 
result will be the use of model outputs by training managers to significantly 
increase the number of alternatives considered in arriving at resource decisions. 
The overall resultant will be more efficient use of training resources. 

The three DOTS models were selected as representing a cross-section of the 
significant planning decisions addressable by computerized mathematical models. 
Using the selected models as test vehicles, the T&E of Phase III was concerned 
with their evaluation within two problem areas: 

• NAVEDTRACOM Decision Process 

The first problem area was concerned with the control and planning 
of requirements and resources. The Department of Defense (DoD) Pro¬ 
gramming, Planning and Budgeting System (PPBS) is the primary 
NAVEDTRACOM driver. As the result of various PPBS actions, the 
NAVEDTRACOM resource planning and control process is directly 
impacted with appropriate management actions being initiated at 
multiple levels. These "appropriate management actions" are 
based on information flowing from both lower and higher levels of 
organization. 

The T&E team was tasked with determining the degree of capability 
of the DOTS models for assisting the decision process within the 
framework of this system. In addition, recommendations as to 
level and techniques of application were to be developed. 

• NAVEDTRACOM Operational Environment 

The second problem area was concerned with the practical Implications 
of implementing computerized mathematical models in terms of finan¬ 
cial feasibility and the operational Impact on support personnel. 

The T&E was tasked with evaluating the DOTS models based on observa¬ 
tions as they were operated in the Norfolk FLETRACEN test bed. 

Within the context of these two problem areas, the Phase III T&E provided guid¬ 
ance to the future direction of DOTS as well as to an assessment of the three 
models. The T&E results are summarized in SECTION III, TEST AND EVALUATION 
RESULTS. 


1-3 
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SECTION II 

TEST AND EVALUATION APPROACH 


OBJECTIVE 

The T&E team objective was a very significant one in relation to the overall 
DOTS' R&D effort as represented by Phases I, II and III. Through the team's 
assessment of the three DOTS models, conclusions and recommendations were 
drawn pertinent to the two problem areas defined under SECTION I's PROBLEM 
STATEMENT subsection. The T&E results also provided a basis for making more 
general recommendations pertinent to the application of managerial science, 
operations research, and mathematical modeling techniques to the NAVEDTRACOM 
planning and decision process. 

In support of the overall DOTS' objective, Phase Ill's major tasks were: 

• To develop and document conclusions pertinent to the DOTS' hypothesis 
which states that the application of managerial science, operations 
research, and mathematical modeling techniques could significantly 
enhance the planning and decision process within the NAVEDTRACOM 
resulting in a significant gain in efficiency. 

• To develop recommendations giving direction to future DOTS' R&D 
efforts as well as possible operational implementation of DOTS 
type models. 

Using the outputs of Phase II, validated test models and a selected test bed. 
Phase III activities of the IBM technical team required to achieve the above 
were: 

• To design and develop the unique software required to support 
implementation of the DOTS' test models at the test bed, Norfolk 
Fleet Training Center (FLETRACEN). 

• To accomplish orientation and training of Fleet Training Center 
personnel and members of the Government Test and Evaluation Team. 

• To develop sufficient technical documentation to permit continued 
development or modification of the DOTS' models beyond Phase III. 

The activities of the Government T&E Team are implied by the Problem Statements 
previously described. The Team's objective was to provide an evaluation and/or 
judgmental assessment of the :(in order of descending priority) 

• Potential of the DOTS type mathematical model and the systems 
approach for enhancing the management of resource and requirements 
within the total NAVEDTRACOM. 

• Technical validity of the three test models. 

• Value of the model's capabilities, assuming reasonable modifications 
to the present management system. 
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t Operational practicality and financial implications of the DOTS models 
operating in the Norfolk FLETRACEN. 

Based on the T&E, conclusions and recommendations were developed and are summar 
ized in this report. The potentially operational by-products of the testing 
process were documented in two separate volumes: a User's Manual and a Program 
Maintenance Manual. These are described in Section IV of this document. 
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SUPPORTING TASKS 

To give perspective to the Phase III tasks, this subsection will present an 
overview of the DOTS' tasks accomplished during Phases I and II but pertinent 
to the Phase III T&E. The following chart will assist In maintaining a time 
reference. 


TASK DESCRIPTION 


0 > 


M 


M 


PHASE I 

• Functionally Analyze the NAVEDTRACOM 

• Develop a Functional Model 

’• Define the NAVEDTRACOM Environment of the 1980's 

• Define Gaps and Problem Areas 

• Develop Recommendations 


> 


PHASE II 

► # Develop Long Range Implications 


Develop Selected Test Models 


Validate Test Models 


• Develop Test Modal Documentation 


PHASE III 

• Develop Test Bed Data Base 

• Develop Test Bed Software 



• Develop User's Guide 


• Support the T and E 


• Develop Final Documentation 



# Identify Potential 
Modeling 
Candidates 


Select Test Mod'h 


wVWCl 

Operational 
Test Bad 


PHASE II A 

• Implement Test Bed 

— S elect Time Sharing Service 
— Install Test Terminal 
- Install Test Models 

• Develop Training Program 



Test 

and 

Evaluation 


# Conduct 
Training 


DOTS Test & Evaluation Supporting Tasks 
Phases 1,11 and III 
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T&E, Phases I and II Supporting Tasks (Cont'd) 


The T&E related tasks were: 

• Select Test Models (Phases I and II) 

The details of the test model selection process are covered in the 
Phase I Report 4 and will not be repeated here. The three selected 
models, Educational Technology Evaluation (ETE), Training Process 
Flow (TPF) and System Capabilities/Requirements and Resources 
(SCRR), met two selection principles of permitting an assessment 
of the DOTS hypothesis, amenable to extrapolative analysis, and 
of providing a potential operational capability. The greatest 
weight was given to selecting models most supportive of the 
DOTS test objectives. 

The selected test models are intended to be complementary. Three 
models, as opposed to one, were selected to facilitate DOTS hypothe 
sis evaluation across a broader spectrum of operation, as well as 
of multi-level application. Although none of the three test models 
is time restricted, each has been designed to emphasize a range of 
operational, near horizon, or long-range planning. The types of 
requirements addressed by the test models were common to all levels 
of the NAVEDTRACOM and, therefore, extrapolation of results beyond 
the actual test bed was feasible. 

• Develop Test Models (Phase II) 

The three test models were developed during Phase II to an opera¬ 
tional test installation level and are described in the Phase II 
Final Report 5 . The designs were based on operational needs identi¬ 
fied during Phases I and II. 

• Validate Test Models (Phase II) 

The purpose of the validation task was to objectively demonstrate 
that all sub-elements of the three test models would perform the 
functions they were designed to do. In short, did the models 
logically perform as intended? 

The validation was performed in a laboratory environment, i.e., 
validation runs were made on a computer and with data remote from 
the ultimate test bed. Validation should not be confused with 
verification. Validation proves the basic design logic of a model, 
verification proves the accuracy of a model's predictions or its 
ability to simulate a process or environment. 


^ Design of Training Systems, Phase I Final Report , TAEG Report No. 12-1, 
December 1973, Pages VII-1 through VI1-33. 

5 Pesign of Training Systems, Phase II Final Report , TAEG Report No. 12-2, 
Vols. 2 and 3, December 1974. 
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T&E, Phases I and II Supporting Tasks (Cont'd) 


Phase II's validation task is described in the Phase II Final 
Report®. The three test models were successfully validated. 

• Select T&E Test Bed (Phase II) 

Based on the original DOTS' plan. Phase III verification was to have 
taken place in the same laboratory environment as the Phase II 
validation. To increase accuracy, it was decided to actually per¬ 
form this task in an operational test bed. The approach offered 
a secondary benefit; if the models did demonstrate utility, the 
potential for more immediate practical application would be 
enhanced. 

The change to an operational test bed also permitted expansion 
of Phase Ill's scope to encompass potential human factors pertinent 
to operational use as well as technical verification of the test 
model's fidelity. 

Based on the empirical observations of Phase I, there seemed to be 
little doubt that mathematical models could be developed providing 
valid predictive information for the NAVEDTRACOM planning and 
decision process. Phase II's validation of the three test models 
subsequently offered empirical support to this supposition. The 
most difficult issue had to do with the operational implications of 
various management levels actually using the models within the existing 
organizational, procedural and motivational system. This issue spanned 
a wide range of concerns from the practical data processing support 
required to the support of DOTS concepts by various management levels. 

Plans for Phase Ill's formal T&E were changed to permit evaluation 
in an operational test bed rather than experimentally in a labora¬ 
tory environment. By making use of an existing commercial time¬ 
sharing service and installing a teleprocessing display terminal 
and printer in an operational test bed, the formal T&E evaluation 
could be based on observation in an operating environment closely 
resembling a projected real-world situation. 

Based on the above, a task was added to Phase II to select a 
viable operational test bed. The Norfolk FLETRACEN was selected 
from a number considered since it most closely matched the entire 
span of considerations and was of a size permitting installation 
of the DOTS' models within the scope of available resources. 

More specifically, Norfolk: 

- Had an extensive mix of "A" and "C" schools. 

- Supported a fairly broad combination of ratings. 


^ Design of Training Systems, Phase II Final Report . TAEG Report No. 12-2, 
Vol. 2, December 1974, Pages 11-30-60, III-19-32 and IV-22-32. 
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T&E, Phases I and II Supporting Tasks (Cont'd) 


- Was subjected to unpredicted requirements for new courses, 
new quotas, etc. 

- Was responsible for all functions in the training development 
cycle between course design and course implementation. 

- Was involved in the implementation or planning of new 
instructional techniques such as Individualized Learning. 

- Had acceptance of scientific management techniques at the 
command level and a desire to participate in the DOTS' T&E. 

- Was general enough in level and type to be representative 
of a significant fraction of Navy training. 

- Was of a size, permitting development of a data base within 
the scope of the DOTS' effort. 

The three test models were selected based on their potential for 
evaluating the original DOTS' hypothesis and not for their immediate 
value to the Norfolk FLETRACEN. To partially compensate Norfolk 
for its interest and effort in supporting the test and evaluation, 
some limited software was developed to permit access to the DOTS 
supplemental data base required to drive the test models. 

The selection, development, and logic validation of the test models, 
and the operational test bed selection, were accomplished during 
Phase II. The following tasks supportive of the T&E were completed 
during Phase III: 

• Test Bed Implementation 

This task included selection of a time-sharing service and 
supporting terminal, development of Fleet Training Center, 
Norfolk (FLETRACEN, NORVA), unique software and data base, 
training of operational personnel, and initial operational 
support of model use. 

• Support Navy Test and Evaluation 

The Navy T&E team was tasked with the actual T&E. IBM was 
tasked with the development of the T&E plan and a training 
program for the T&E team members. 

• Support Documentation 

IBM developed sufficient documentation during Phase III to 
assure continuity of the project, whether the Intent was 
additional R&D or projects leading to operational imple¬ 
mentation of the models. These documents are described 
in SECTION IV, PHASE III PRODUCTS. 
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T&E, Phases I and II Supporting Tasks (Cont'd) 


• T&E Results Analysis 

The Government T&E Team devoted considerable effort and 
time, after their Norfolk training and test, to results 
analysis and documentation. Its output was a Final 
Report which is summarized in SECTION III, TEST AND 
EVALUATION, TEST AND EVALUATION SUMMARY. 
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TEST AND EVALUATION 


The Government T&E Team was formalized In April of 1975 and consisted of: 


F. DiGialleonardo - Team Leader - NPRDC - Washington 
LCDR R. J. Blersner, Code N-214 - CNETS - Pensacola 
Lt. R. N. Brooks - TRALANT - Norfolk 
CDR J. D. Davis, Code 015 - CNTECHTRA - Memphis 
LCDR T. L. Ferrler, Code N-31 - TRAPAC - San Diego 
Mr. J. T. Finnlgan, Code N-31 - TRAPAC - San Diego 
Mr. E. Scheye, Code N-336 - CNET - Pensacola 
Mr. D. S. Thomas, N-215 - CNTECHTRA - Memphis 


The T&E Team was supported by the 

Mr. William Lindahl - TAEG 
Mr. Thomas McNaney - TAEG 
Mr. Harold Bellamy - IBM 
Mr. Kenneth Branch - IBM 
Mr. Charles Edison - IBM 
Mr. J. D. Staley - IBM 
Mr. Ronald Yanko - IBM 


following IBM and TAEG personnel: 


- Orlando 

- Orlando 

- Cape Canaveral 

- Cape Canaveral 

- Cape Canaveral 

- Cape Canaveral 


In April, the T&E team was provided an IBM-developed test plan^ and goals. 
However, under direction of the team leader, a unique plan and goals were 
developed prior to Initiation of the formal T&E. Generally, the differences 
were In degree of emphasis rather than In areas and concepts to be tested. 


Both plans were based on the results of a joint Norfolk FLETRACEN/TAEG/IBM 
problem Identification workshop held In December of 1974. After being intro¬ 
duced to the problem solving capabilities of the DOTS models, requirements and 
problem statements were solicited from FLETRACEN management personnel. The 
solicitation was not restricted to only those problems addressed by the models. 
Responses ranged from requests for specific data items to requirements for 
sophisticated analysis of interrelationships between personnel characteristics 
and academic potential. The problems Identified were concerned with such areas as 


• Impact of student load changes on instructor requirements. 

• Relationship of contact hours to the number of instructors. 

• Impact of various factors on the attrition rate. 

• Correlation of instructor capabilities to potential cross-training. 

• Relationship of personnel cuts to student load. 

• Relationship of convening frequency to such areas as utilization, 
capacity, convenience, attrltltlon, etc. 


7 pes1gn of Training Systems, Preliminary User's Test Guide . TAEG, Section III, 
31 March 1975. ' 
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Test and Evaluation (Cont'd) 


The schedule of formal Phase III T&E events was as follows: 

• 4 June through 6 June 1975: Training by IBM of the Norfolk 
FLETRACEN staff. 

• 16 June through 20 June 1975: Training by IBM of the T&E team. 

• 23 June through 27 June 1975: T&E team model analysis at the 
Norfolk FLETRACEN. 

• 30 June through 5 September 1975: Visits to various functional com¬ 
mands by the T&E team leader, accompanied by a member of the Project 
Office, and documentation of the final results. 

The approach used during each of these phases was as follows: 

• Training at the Norfolk FLETRACEN Staff 

Training course design was based on the assumption that there were 
three types of personnel concerned with model application. These 
were the executive or decision-making level, the analyst who 
could provide the model capability interface between executive 
and system, and the operator who would actually perform mechanical 
interface operations such as data update or calling forth the 
models. These three levels did not include the responsibility of 
technical maintenance of the models since it was assumed that, if 
the models were subsequently Implemented operationally, maintenance 
would take place at a central location remote from the operational 
user. 

The executive level received a limited formal introduction to the 
DOTS system with the objective of simply creating an awareness 
of the DOTS capability. 

The analyst level received two days of intensive orientation. 

The emphasis was on what the models would do rather than how 
they did it. Operator training concentrated on the more mechani¬ 
cal aspects of system manipulation. It should be remembered that 
the purpose of training the Norfolk personnel was to permit the 
T&E team to assess the operational feasibility of implementation 
of DOTS type system in a "real-world" environment and using 
existing operational personnel. 

The original Preliminary User's Test Guide was significantly 
expanded to be used in support of this training as well as the 
subsequent T&E. This final User's Test Guide* * included sections on the 
mechanical operation of the models, diagnostic validation tests for 


8 Pes1qn of Training Systems, User's Test Guide . TAEG, 30 May 1975. 
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Test and Evaluation (Cont'd) 


each model to permit revalidation, and projected cost factors pertinent 
to using the models. 

• Training of the T&E Team 

As previously indicated, the team was provided information in the form of 
a preliminary guide and goals. In addition, prior to formal T&E training, 
the team members were provided comprehensive guidelines by the team leader. 
These guidelines will be summarized in the next sub-section. The team 
entered the formal training session at Norfolk with an understanding 
of these documents. 

During formal sessions, the T&E team members were given executive, 
analyst, and some operator level training. This broader scope was 
included to permit a more valid assessment of the operational consid¬ 
erations to be evaluated. 

• T&E Team Model Analysis 

The week following T&E training was devoted to analysis at Norfolk. 

The IBM team was available for technical assistance during this week. 

The T&E team analysis was organized around three major assessment 
areas. These were operational, technical, and financial feasibility 
of the DOTS models. It was understood that the main emphasis of the week 
of analysis would be on operational feasibility, although Initial 
data for the technical and financial areas would be identified. ^ 

To accomplish the above, the T&E team was divided into five dis¬ 
cussion areas, each managed by a leader. These areas and their 
sub-areas were as follows: 

Potential Model Contributions 


At the activity level 
At other levels 

Ease and Practicality of Use 

User knowledge requirements 
Update requirements 
Response time 
Output form 

Data Requirements 

Nature of the data utilized by models 
Data availability 
Data accessibility 
Data management 
Future data problems 
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Test and Evaluation (Cont'd) 


Organizational Implementation 
Prospective levels of application 

Relationship of models to existing policies and structures 
Implementation support 

User Investment 


Personnel resources 
Hardware 

Method of operation 

The T&E team completed their analysis week with a preliminary posi¬ 
tion that was refined and expanded during the follow-on analysis to 
be described in the next sub-section. 

• Functional Visitation 

Following the week of analysis at Norfolk, and extending to September, 
the various members of the T&E team and TAEG conducted a number of 

functional visitations and reviews pertinent to the preliminary 

results of the June analysis. 

Final documentation^ was completed during this period. A summary 
and selected excerpts from this final report are covered in 

the next SECTION III. The T&E Report should be considered as one of 

the major products of the three-phase DOTS' effort since it repre¬ 
sents an assessment of the practical value of mathematical modeling 
techniques to the NAVEDTRACOM. 

In order to accomplish the T&E at Norfolk, it was necessary to provide the 
Norfolk FLETRACEN with access to a computer. A Command and Control Center 
(CCC) was installed consisting of an alpha/numeric display terminal and 
printer linked via dial-up telephone lines to a commercial time-sharing 
computer service. National CSS, Inc., located in Norwalk, Connecticut. NCSS 
provided the following equipment at Norfolk: 

• One Hazel tine 2000 Video Display Terminal. The Hazel tine 2000 
displays 25 lines of 80 characters each and operates at a 30 cps 
transmission rate. 

• One Hazel tine Impact Printer Model 300. The Model 300 prints at 
30 cps, has a carriage width of 118 characters, and employs a 
tractor-type paper feed. 

• One Hazel tine Dual Magnetic Tape Cassette Unit, having a 400 bpi 
storage density and a capacity of approximately 125,000 characters. 

The Cassette Unit, which can operate in either an off-line or on¬ 
line mode, serves as a storage unit for later display of data on 

the Video Display Terminal and for editing text inputs to the system. 

9 fe C °Kug; SeptS n |^! ra1n1nq Sy,te "’ Test and Eva1uat1nn Fina1 
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Test and Evaluation (Cont'd) 


The CCC configuration was intended to provide a full complement of terminal 
capabilities for evaluation of model usability in an operational environment. 
The NCSS accounting system also provided specific costs for each interaction 
as input to the financial feasibility assessment of the T&E. 

In summary, the T&E was directed toward the assessment of a wide range of 
considerations, as opposed to simple verification of model design. As 
previously indicated, it could be assumed that models could be identified, 
designed, and developed that would work. The key question had to do with 
the practicality of models in the unique NAVEDTRACOM environment. 

The T&E team did not assess the various sub-task results, Phases I and II, 
leading to test model selection or general recommendations. The T&E Final 
Report specifically excludes the following: 

• Comprehensive functional study of NAVEDTRACOM. 

• Strategic assumptions for the 1980 decade. 

• Delineation of technological gaps in the training system. 

• Development of the Educational Technology Assessment Model (ETAM). 

• Audio-Visual aids for orienting Navy training managers to the 
nature and use of management science techniques. 

The first three items are covered in the Phase I Final Report; the fourth 
(ETAM) is currently under development; the fifth is a set of tape/slide 
presentations providing an orientation tutorial pertinent to the three 
DOTS models. 


11-12 






r 


TAEG REPORT NO. 28 


SECTION III 


TEST AND EVALUATION RESULTS 


TEST AND EVALUATION SUMMARY 

The purpose of this section is to provide a summary of the DOTS 1 Test and 
Evaluation, Final Report 1 0 and IBM comments pertinent to the T&E's positions. 

A summary may highlight key conclusions but it cannot substitute for an in-depth 
review of the actual report. Although every attempt was made to avoid bias 
in this summarization, it is possible that the intended emphasis was inadvertently 
shifted. As appropriate, direct quotes from the T&E Report will be included. In 
every case, these quotes are outlined in a separate box. 

The T&E Final Report is organized as follows: 

• Part One - Summary of Results and Recommendations 

• Part Tv/o - Technical Feasibility 

• Part Three - Operational Feasibility 
t Part Four - Financial Feasibility 

• Appendices - A through G 

Generally, this summary will follow the same format. Abbreviated summary 
statements will be followed by selected quotes and the quotes by a commentary. 
Report page numbers are provided. 

All members of the IBM technical team reviewed the T&E Report and were given 
an opportunity to respond to its conclusions. Generally, there was agreement 
with the conclusions and commendation for the report's thoroughness and pro¬ 
fessional ism. 

The chart on the next page provides a summary of the T&E Statements as well as 
SECTION III page locations. 


l°DiGialleonardo, F., Design of Training Systems, Test and Evaluation Final Report, 

NPRDC, September 1975. 
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DESIGN OF TRAINING SYSTEMS 

PHASE III TEST AND EVALUATION SUMMARY STATEMENTS 



QUOTATIONS 
SECTION III 


IBM 

COMMENTARY 

From 

Page 

To 

Page 

T&E SUMMARY STATEMENTS 

From 

Page 

To 

Page 

III 3 

1114 

General Conclusion.Considering technical, operational and financial factors, it is feasible to apply 

mathematical modeling techniques to the NAVEDTRACOM planning and decision process. 

1114 

MI-5 

MI-6 

111-9 

Technical Feasibility-The three DOTS models are logically valid (function as designed) and feasible. 

It was not proven (nor disproven) that they accurately simulate or predict actual situations. 

MI-9 

lll-IO 

111*1 1 

111-17 

Operational Feasibility-.The DOTS models are operationally feasible at the schoolhouse and func¬ 

tional command (COMTRALANT and COMTRAPAC) levels considering operational personnel required 
for implementation, current management structure and practices, existing operational and planned 
automated data systems, and DOTS data requirements. 

MI-18 

MI-18 

111-19 

MI-21 

Financial Feasibility-The three DOTS models were not proven (nor disproven) to be financially 

feasible, based on either current operational tasks displaced or costs avoided through model applica¬ 
tion. Based on empirical observation, costs of implementation appear to be minimal and, therefore, 
could probably be justified on the basis of probable benefits. 

IM-21 

MI-22 

111-24 

111-25 

Recommendations-.While the DOTS effort appears to be complementary to existing and planned 

NAVEDTRACOM applications of Automated Data Systems (ADS), there is a need for significantly 
increased management and coordination of the composite ADS by CNET. This increase must take 
place if maximum benefit is going to be derived from any data base or modeling effort. 

111-25 

MI-28 

III 29 

III 30 

Criticism .The three DOTS models were not verified during Phase III. 

MI-30 

111-31 

IM-32 

; n-33 

Criticism ——Utility of the DOTS test models has not been proven. 

MI-33 

MI-33 

11-34 

111-35 

Recommendation -.Maximum utility of mathematical models can only be achieved through applica- 

tion to higher command levels than that represented by the Norfolk Fleet Training Center. 

III-3S 

IM-38 

111-39 

111-39 

Criticism -.The Phase II benefit/cost analysis is not valid. 

111-39 

11143 

111-44 

III 44 

Recommendation -There is a need for establishment of a NAVEDTRACOM wide program, under 

direction of a CNET agent, devoted to the development and coordination of management science 
techniques. 

11144 

11145 

ill-46 

11146 

Criticism-The cost of developing the three test models appears excessive. 

11146 

11146 
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• General Conclusion (Part 1 - T&E) 

Statement - General Conclusion 

Considering technical, operational and financial factors, it is 
feasible to apply mathematical modeling techniques to the NAVEDTRACOM 
planning and decision process. 

Quotes: (T&E Report) 

Page 5 .. .It is the conclusion of this analyst that the DOTS 
models have established beyond reasonable doubt the feasibility 
of applying modeling techniques to Navy training management 
problems. This general finding is a composite of the elements 
of technical, operational, and economic feasibility... 

Page 6 .. .The SCRR model demonstrates the technical feasibility 
of providing training center managers with more accurate and 
multi-dimensional assessments of their training capabilities ... 

...In its current non-statistical form, the TPF has shown that 
it can be a useful tool in the hands of training managers by 
taking full advantage of presently available data and certain 
operationally utilized, if still ill-defined, planning factors 
extant in the current training environment... 

Page ...No particular faults could be found with the composi¬ 
tion of the ETE. Its primary problem is that while none of the 
models were strictly field tested, the ETE was the least tested 
of the lot... 

...The results of this validation, while thoroughly documented, 
do not provide insights to how useful the model can be to actual 
users... 

...This resulted in substantial proof that the ETE, despite its 
generality, could perform in superior fashion to a system 
specific model in a given case. It does not tell anything 
about whether there will be much opportunity for use of such a 
model given the nature of the EDTRACOM 9 s operations. A more 
meaningful test of the appropriateness of the ETE to current 
or future Navy training problems was apparently limited by the 
unavailability of appropriate subject courses... 

Page 10 ...Opportunities for applying the models at the TRACOM 
level look exceedingly good. Both COMTRALANT and COMTRAPAC 
seem receptive to application of the models in their respective 
organizations... 

Page 50 ...The potential for contribution by the DOTS models 
in the overall planning area is its most significant feature... 
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Statement - General Conclusion (Cont'd) 

...DOTS is envisioned as a possible device to effect the 
changes required in these various systems to provide a truly 
reliable and accurate MIS with the added ability to model... 

...If the DOTS concept did nothing more than precipitate an 
investigation and realignment of the current morass of Train¬ 
ing Command MIS(s) it would be of value. Given that it will 
provide capabilities to model and simulate and therefore pro¬ 
ject as well, it is potentially of great value... 


Commentary: 

The quotes above provide positive support to the General Conclusion Statement. 
However, they should be tempered with those included in the Criticism and 
Recommendation sub-section, which will follow the next three sub-sections. 

The T&E supported the following conclusions from the DOTS Phase II Final 
Report^!: 

Conclusions (Quote from the Phase II Final Report) 

The Naval Education and Training System can improve the effectiveness 
of its decisions pertinent to resource planning and control through 
use of computerized mathematical models... 

The SCRR, ETE, and TPF models are logically valid and do perform 
as designed... 

Sufficient historical and operational data are available in existing 
records to enable operational implementation on the DOTS' models... 

As will be repeatedly stressed in this SECTION, the T&E team was charged with 
assessing the DOTS' test models as they apply to the current NAVEDTRACOM deci¬ 
sion and planning process. The T&E team's endorsement should not be extrapo¬ 
lated to include the DOTS Phase II conclusion that, to gain maximum benefit 
from mathematical modeling, changes will be required in the decision and plan¬ 
ning process. The Phase II Final Report stated, "This can be accomplished 
with reasonable changes to the current management system and practices." 
SECTION IV of this report will address the question of change. 

Also, the endorsement of model logic and design should not be interpreted as 
applying to model fidelity. The T&E team did not feel that the ability of the 
models to accurately simulate or predict actual events had been proven or 
disproven. The subject of historical verification will be covered in the next 
sub-section concerning Technical Feasibility. 


!! Design of Training Systems, Phase II Final Report, TAEG Report No. 12-2, Vol. 1, 
December 1974, Page V-l. " 
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Statement - General Conclusion - Commentary (Cont'd) 


The T&E team approached its evaluation on the assumption the three DOTS' test 
models would be applied within the existing planning and decision structure, 
procedures, and motivational environment. The original test model and bed 
selections were made on the assumption that the existing structure, procedures 
and motivational environment would be subjected to "reasonable change." This 
dichotomy should be considered in reviewing the T&E results. 
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• Technical Feasibility (Part 2 - T&E) 

Statement - Technical Feasibility 

The three DOTS models are logically valid (function as designed) 
and feasible. It was not proven (nor disproven) that they 
accurately simulate or predict actual situations. 

Quotes: (T&E Final Report) 

SCRR 


Page 16 ...The SCRR was developed to meet four specific 
types of management problems. These are discussed below... 

.. . Assessment of io£2z£2£E $£**£±22. ^ eman< ^ • • •What it can 
do is take near or long-term training demand projections 
as input, and then operate to determine whether such require¬ 
ments can be feasibly met given current or projected resources 
and procedures. In this sense, given sufficient lead time, 
it can aid in training resource planning and capabilities 
assessment... 

Page 17 ..‘ Assessment of short-term demand fluctuation arising 
from unscheduled events (e.g ., a_ Ship repaip operation. , reserve 
activation , unusual seasonal recruitment levels) . Here 
different resource and demand configurations can be tried out 
to assess the impact of such perturbations, minimizing their 
dysfunctional effects and maximizing the responsiveness of 
training, through better allocation of available resources... 

.. . Assessment of training resource utilization ...Using 
current demand, SCRR can compute an optimum resource combina¬ 
tion ... 

... Comparison of alternative resource allocation strategieg ... 
This is a case of the model in which multiple strategies are 
evaluated, given known performance and requirements character¬ 
istics. This is one of the best used of the linear program¬ 
ming capabilities of the models and could precipitate substan¬ 
tive changes in present procedures for developing plans and 
programs as well as justifying and revising the same... 

Page 21 ... Model Validation ...The basic LP technology and 

the MPSX software are, of course, not in need of validation 
due to their standardized nature. The validation documented 
by the model developer (TAEG Report 12 - 2 , Vol. II, pp 11 - 30 - 60 ) 
demonstrate that the model is functioning properly. The 
model's ability to accurately represent the real world 
is not demonstrated. This must be done with historical data 
or under field test conditions. In the absence of such 
information, the SCRR is judged to be a useful and practical 
representation of the training complex... 
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Statement - Technical Feasibility (Cont'd) 

...This judgment is a function of (1) feedback from pro¬ 
spective users obtained in the user TSE and other discus¬ 
sions, (2) a comparison of the model's formulation with 
a description of the training complex operation... 

TPF 

Page 22 ...The objective of the TPF is to provide a means for 
simulating the flow of students through a training complex... 

...Given inputs such as maximum or average course capacity, 
number of convenings, and demand, the TPF computes course 
utilization, backlog, AOB, etc., showing these on a quar¬ 
terly basis through time... 

...The variables currently dealt with by the TPF have been 
screened from a larger set of variables on the basis of 
statistical analyses. An original objective of the TPF 
was to develop statistical parameters from these analyses 
which could be included in the model for the purpose of 
predicting training results as a function of student 
profile data. Correlations resulting from these statis¬ 
tical analyses were not of sufficient significance to 
warrant inclusion in the model and therefore were not 
incorporated... 

...the TPF, as a simulation, mathematically represents the 
mechanics of the training center scheduling function from a 
deterministic rather than probabilistic basis... 

...The magnitude of original potential payoff from this 
model is substantially reduced as a result of its present 
inability to statistically relate student profile data to 
student behavior within the training system and the conse¬ 
quent inability to gauge the potential impact of such 
factors... 

Page 24 .. .student profile data and the results of the 
statistical analysis of that kind of data are not incor¬ 
porated into the model... 

...the idea of formulating a model to predict impact on the 
training process from various types of student inputs remains 
intuitively desirable... 

Page 25 .. . Model Validation . The validation procedures for 
the TPF as described by the model developer (TAEG Report 12-2, 
Vol. II, IV, pp 22-30) do not constitute a true validation. 

The model is simply exercised in a number of conditions and 
its deterministic processes are allowed to respond accordingly. 
It is clearly demonstrated that the model is in operating order. 
However, there is no basis for judging how well the model 
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Statement - Technical Feasibility (Cont'd) 


simulates the real training process flow. All that can be 
said is that the factors utilized in the TPF appear to be 
reasonable since they are drawn from those which are commonly 
used by the training managers... 

...Thus we know that the TPF is a useful automated version of 
the kind of scheduling that is presently being carried out 
at the FTC. We have a more flexible and quick means for 
developing schedules and assessing demand impacts. It has 
not been demonstrated that we have a more precise or accurate 
means for making such developments or assessments... 

ETE 

Page 26 ...The ETE is an entity flow simulation model for 
ILS course design and management. Like the TPF and SCRR, 
this model is intended to answer questions concerning 
capabilities (capacity), utilization, and resource require¬ 
ments as well as course completion time. It has the capability 
to evaluate, in compressed time, the effects of real or pro¬ 
jected changes in resources or demand. Since equipment and 
personnel can be costed external to the model, the model can 
be used in making cost-effectiveness determinations of dif¬ 
ferent ILS course designs... 

Page 27 ...The validation was not in terms of how well the 
ETE simulation represented actual course behavior, but 
rather to the extent that results replicated those obtained 
from the application of a previously developed special 
purpose simulation model of the EW school... 

...it is only known that the ETE is as good as or better than 
another simulation model as applied to the EW school use. Nor 
does this validation procedure establish the generality of the 
ETE as implied by the model developer (TAEG Report 12-2, 

Vol. II, pp 23-24)... 

...With the exception of the model validation procedure 
(which may have been a necessity) the ETE appears to have 
been soundly developed from a technical viewpoint. However, 
this technical soundness is as of yet, not bulwarked by a 
clearly demonstrated utility (i.e., a utility determination 
arrived at through interaction with actual ILS designers and 
managers)... 


General 


Page 28 ...In general, the quality of the validation proce¬ 
dures for the DOTS models conducted during both model develop¬ 
ment and the T&E, suffers from the lack of a legitimate field 
test... 
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Statement - Technical Feasibility (Cont'd) 


...Present conditions for field testing of the SCRR and TPF 
appear to be very good at both the COMTRALANT and COMTRAPAC 
sites. Furthermore, prospects for usable end products 
emerging from such testing are sizable... 


Commentary: 

The three DOTS models were designed and developed by three individuals. Each re¬ 
viewed the T&E Final Report and, although there was limited disagreement on some 
of the Report's minor positions, the modelers felt the key points and conclusions 
were accurate and thoroughly assessed. 

As was indicated in the T&E Report, the Team evaluated the models against their 
application within the current organization and procedural process for planning 
and control of resources and requirements. Although the majority of the T&E was 
concentrated on a pragmatic analysis of the models as operational tools, some 
attention was given to their assessment as subsets to test the DOTS' recommenda¬ 
tions evolving from Phases I and Il12. The following quotes are presented in 
the Commentary sub-section since, to a certain degree, they address those DOTS' 
recommendations pertinent to higher levels of models and institutional change. 


Quotes: (T&E Final Report) 


Page 7 _ .. .The SCRR and TPF models were developed in a fleet 
training environment. Here the demand is characteristically 
unprogrammed. It is reasonable to believe that in other 
Navy training sectors where demand is much more programmable 
(e.g., technical training), the potential for effective 
model development is at least as great if not greater than 
has been demonstrated for the fleet environment... 

Page 9 .. . The models themselves neither require nor warrant 
such incipient aggrandizement. They are good examples of 
how basic management science technologies can be utilized 
to assist in the solution of specific management problems . 

As such, they imply that establishment of a program, 
chartered to utilize management science techniques in the 
support of Navy training management would be well founded. 
Such a program should not operate on the basic assumption 
that there is a total integrated solution to the training 
management problem, but rather on the objective of achieving 
incremental improvements in management effectiveness by pro¬ 
viding useful solutions to important management problems, 
regardless of their limited generality ... 


T7-7 

Design of Training Systems, Phase II Final Report , TAEG Report No. 12-2, 
Vol. 1, December 1974, Page V-l. 
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Statement - Technical Feasibility - Commentary (Cont'd) 


Page 32 ... In addition to more effective performance of the 
training management function as currently practiced, the 
models have the potential for introducing new management 
techniques and supporting certain types of decision making 
not currently practiced. The DOTS developments are seen 
as contributing in two distinct but interdependent modes. 
First, as_ a_ manayernent information system by virtue of 
the data base management capabilities developed to support 
the models. This was not a direct objective of DOTS but 
was a prerequisite to producing models which can effectively 
function in the operational environment. Second, as a 
management decision suyyort system to be used in the kinds 
of functions noted above. More specific evaluation of 
potential model contributions is presented further below ... 

Page 36 ...Several participants viewed the bulk of model 
potential to lie in application at the "functional" level 
and above (i.e., "models and necessary support may not be 
worth it at schoolhouse level"). On the other hand, 
participants with "schoolhouse" level experience expressed 
the view that considerable opportunity exists to utilize 
model potential, given "a little managerial imagination." 

A concise assessment of actual contribution can only be 
obtained via a field test (which FTC, Norfolk, was not)... 

...A possible hindrance to vertical expansion of model 
application is the plethora of overlapping management 
systems which reside at the upper management levels. While 
the integration of these systems is not within the scope 
of DOTS, higher level implementation of DOTS development 
may be jeopardized nevertheless. Coping with this 
dilemma remains a future challenge to the DOTS project 
management... 

Page 43 .. .DOTS is not primarily an information collecting/ 
collating system, but instead provides optimization of 
information output from other systems (i.e., NITRAS). The 
three DOTS models evaluated should be viewed as subsystems 
within a TRACOM MIS... 


Within the boundaries of the T&E, technical feasibility was established. As 
indicated previously, this statement should be tempered with those included 
in the Criticism and Recommendation which concludes this SECTION. 

Except as reflected in the quotes above, it should be emphasized that the 
T&E did not assess the implications of an integrated management system, nor 
was the team tasked with this responsibility. Therefore, endorsement of 
technical feasibility by the T&E should not be interpreted as including all 
of the main points of the DOTS Phase II Conclusions and Recownendations. 
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• Operational Feasibility (Part 3 - T&E) 

Statement - Operational Feasibility 

The DOTS models are operationally feasible at the schoolhouse and 
functional command (COMTRALANT and COMTRAPAC) levels considering 
operational personnel required for implementation, current manage¬ 
ment structure and practices, existing operational and planned 
automated data systems, and DOTS data requirements. 

Quotes: (T&E Final Report) 

SCRR - TPF 

Page 6 ...(SCRR) .. .Like the TPF, it takes advantage of 
existing planning factors such as the student/instructor 
ratios which reflect basic requirements formulas in order 
to determine organizational requirements on the basis of 
the interaction of resources and capabilities. When more 
precise planning factors become available in the future 
(e.g., through the SHORSTAMPS program), they can be 
easily assimilated into the model... 

...The SCRR, in a sense, is an automation of what is 
currently being done mentally by instructors and training 
managers. This is viewed as a plus. Using the SCRR, 
within the course of an hour it might be possible to 
develop as many resource allocation alternatives as a 
training manager could develop in two weeks. This 
advantage, of course, would be multiplied many times 
over considering frequency of occurrence. Add to this 
the capability for optimizing across resource allocation 
alternatives and one can see that the SCRR presents the 
potential for some very real and immediate payoffs for 
the training manager... 

Page 7_ .. .The SCRR and TPF models were developed to be 
consistent with data currently being collected. Inputs 
to the NITRAS data base (which are supplied to CNET) were 
used to comprise a common data base for the two models. 

Certain additional data elements were collected directly 
from the FTC; however, these comprise only a small portion 
of the total data base. All of the data used by these 
models are both available and accessible within CNET. 

About 90% of the data must be collected for reporting to the 
NITRAS, regardless of model requirements. The situation is 
not just peculiar to FTC, Norfolk, but is true throughout 
the CNET commands... 

Page 8 ...In general, members of the User T&E team viewed 
the models as operationally feasible at the TRACOM and 
schoolhouse level. Additional feedback later obtained from 
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Statement - Operational Feasibility (Cont'd) 


COMTRALANT and COMTRAPAC reinforced this finding. However, 
feedback obtained from CNTT claimed that the operational 
feasibility and desirability of the models had not been 
demonstrated at Norfolk... 

Page £ .. .Phases II and III of DOTS have shown that meaning¬ 
ful and useful models can be constructed for the benefit of 
training management. It has also revealed that the most 
effective residence for such models in fleet training may 
be at the TRACOM level... 

...In no way can this line of reasoning be extended to say 
that the models thus far developed are appropriate to CNTT's 
most critical management problems... 

Page 10 . . .Both commands (TRALANT and TRAPAC) have the 
necessary equipment for supporting interactive management 
models... 

...Update requirements for the data base were not excessive... 

...Actual operation of the models was possible with a minimum 
of training. The interactive nature of the model and data 
base programming contributed highly to the ease of use. Ap¬ 
proximately two weeks would be a reasonable time for training 
an operator using basic skill types presently available on 
the organization's staff... 

...Considerably more time would be required to fully educate 
training managers in potential uses of the model, if for no 
other reason than that it would be best to extend such train¬ 
ing overtime thereby allowing real and current problems to 
serve as examples in the training... 

...The present documentation is satisfactory and should serve 
well in future utilization of the models. Though a user's 
guide has been documented, there still remains a need for an 
operator's manual which fully explains model options, error 
messages and corrective measures. It is understood that 
such a manual is presently being prepared by IBM... 

Page 12 ...The time-sharing cost per usage of each model 
is well within reason (User Test Guide Appendix V, 30 May '75). 
Time-sharing, of course, obviates the need for large initial 
investments in ADP hardware, thereby putting the models on 
a largely "pay as go" basis. It does not appear that addi¬ 
tional staff would be necessary to support the models in 
TRACOM operation. Modifications should not prove extensive 
for TRACOM application. All in all, these factors point to 
a very low cost for implementation and operation in the pro¬ 
jected TRACOM application. As a result, it would not require 
very much "benefit" to break even in such a project... 


111-12 









TAEG REPORT NO. 28 


Statement - Operational Feasibility (Cont'd) 

Page 16 ...(SCRR)...The model could be run in a planning mode 
under a number of different demand assumptions. A range of 
training capabilities and resultant impacts could thus be 
assessed... 

...Another product of the SCRR relevant to this aspect is 
the " optimized convening rate ." This is the number of 
course convenings per course that yields maximum student 
throughput...This Optimal Convening Rate would seem to have 
very limited utility... 

Page 17 ...(SCRR)... This usage seems quite desirable given the 
existence of frequent perturbations in demand. Limited usage 
at FTC, Norfolk, seems to indicate that a rather minimal level 
of model analysis can yield Information of broad and recurring 
utility to the training manager. Yet usage of the models by 
managers seems to need at least initial stimulation... 

...The decision to develop an LP model to assess capabilities 
and allocate resources in the training environment was sound... 

Page 19 ...Any misconceptions that a "requirement" model 
introduced to this environment provides better basic require¬ 
ments information than previously available, must be carefully 
avoided. Yet, in the particular case of the SCRR, there is 
little doubt that modeling has demonstrated the potential 
for increasing the quality of management information at the 
FTC as well as for the functional command level above it... 

Page 20 ...The segmentation of resources and requirements 
within the SCRR limits its ability to optimize across the 
training center. While this convention enables most 
candidate FTC problems to be fit within reasonable computer 
storage limitations, it appears to be a less than accurate 
representation of what is possible within the real training 
environment. It may be that some of the greatest management 
payoffs within the FTC are to be gained from optimizing 
resources and requirements across schools... 

Page 22 ...(TPF)... This simulation is intended to enable 
managers to assess the effects of changes in the primary 
variables of the training complex and its immediate environ¬ 
ment. The model operates on two categories of data, those 
pertaining to students (10) and those pertaining to courses 
(15)... 

...Straightforward proportionality figures are currently used 
by the TPF to assess the effects upon the training system of 
changes in demand, scheduling, or capacities. Failure rate, 
no-shows, disenrollments, etc., are not presently forecasted 
by the model on the basis of student profile data... 
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Statement - Operational Feasibility (Cont'd) 


...The choice of a process flow model was a good, if perhaps 
obvious, selection (and as such did not depend greatly on the 
results of Phase I). The course scheduling problem is one 
which both lends itself to modeling and at the same time 
represents a significant training management problem. The 
flow also offers a complementary capability to the resource 
allocation model (SCRR) that was concurrently developed... 

Page 23 .. .Another restrictive factor is the apparently 
limited opportunity which training managers at the FTC 
level have to utilize such a model in a proactive manner. 

This mostly stems from the fact that fleet training demands 
are at the same time hard to predict and hard to deny. On 
the other hand, the more reactive capabilities of the model 
(e.g., to answer "what if" questions) do not appear to be 
in high demand at the FTC level, though they may be peri¬ 
odically of substantial value to that level. It is important 
in considering this to keep in mind the DOTS models were 
selected on the basis demonstrating the feasibility of an 
approach for CNET and not on the basis of what models/ 
techniques would yield the highest payoff to FTC, Norfolk, 
in particular... 

...(SCRR & TPF) .. .answers from either model would be 
partially naive without those of the other. This prompts 
the question of why select and separately develop two models, 
neither of which is sufficient by itself, rather than develop 
one integrated model from the beginning. The answer to this 
question, I believe, lies in the initial orientation of the 
project -it has been technology oriented rather than problem 
oriented (i.e., "test the feasibility of applying new decision¬ 
making technologies"). Developing two separate models pro¬ 
vided a more secure approach to that problem. Furthermore, 
the original concept of the TPF (i.e., to forecast training 
complex impact as a function of student profile data) lent 
it more viability as a separate model concept... 

Page 36 .. .A major consideration in the realization of 
model potential was determined to be the level and manner 
of model use. While operation of the model mechanisms per se 
(e.g., terminal operation) is not complex, use of model results 
requires all of the skills of the experienced training manager. 
Typical model support requirements (e.g., documentation, ease 
of use) were viewed as essential to potential realization. 

(See "Ease of Use" and Organizational Implementation" selec¬ 
tions that follow.)... 

Page 37 ...Specifically, DOTS presents a way for effectively 
utilizing the vast NITRAS data base in a proactive way... 

Pa 9 e 38 •..Training in use of the DOTS model will be needed 
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Statement - Operational Feasibility (Cont'd) 


at all training command levels, though to varying degrees. 
Essentially three levels of training are required. These are 
familiarization for upper management, system analysis for 
translating upper management or command training problems 
into model parameters, and a systems operation for train¬ 
ing those who would operate hardware components of DOTS... 

...In most cases (except at center configuration manage¬ 
ment level), system analysis and system operation would 
be collateral duty for the personnel involved... 

Page 40 ...It was difficult to evaluate this area (Data 
Base Update Requirements) because sufficient new problems 
were not used in the TSE. Program (model) updating/ 
modification would probably be rare for routine (local 
command) report generation. A more extensive program 
update requirement would probably exist at upper manage¬ 
ment levels for resolving projected ("what if") problems... 

Page 42 ...Basic data collection/verification/input to 
the master data base would be responsibilities of local 
commands (centers). These procedures are more acceptable 
than with other current information because local commands 
would have direct access to central system. These com¬ 
mands would also have to manage problem-solving require¬ 
ments unique to the command or center... 

...The TSE Team did not think the programs and data base 
associated with the SCRRSTPF models would be frequently 
updated in the operational phase... 

Page 43 ...The TSE Team believed that neither master data 
base nor program modifications would be difficult if 
proper justification for making these modifications could 
be demonstrated (through findings of RSD or training 
effectiveness evaluations)... 

...Inasmuch as DOTS is primarily a projection/management 
system which provides information for "what if" questions, 
its use as an information/report-generating system would 
be redundant, and would represent gross underusage of 
the system... 

...If used as an information gathering system only, DOTS 
would not substantially improve on response times associ¬ 
ated with other available systems... 

Page 46 ...The core of the data base is NITRAS data which, 
as "raw data," is relatively valid, reliable and accurate... 

(IBM Note: Exceptions were noted in the T&E Report.) 
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Statement - Operational Feasibility (Cont'd) 


...The data are presently available, largely without addi¬ 
tional cost... 

...The data is not presently either easily or regularly 
accessible... (From the DOTS Data Base - extraction pro¬ 
grams required.) 

...Data base is manipulatable in a very impressive way; 
however^, the system for accomplishing this manipulation 
is not Navy owned and is equipment specific... 

Page 47 ...The usability of data to be accessed by the 
DOTS family of models relates directly to the NITRAS/NAVTIS 
problems; such problems as validity, accessibility, avail¬ 
ability, etc., hinge on NITRAS and other systems (e.g., 
SHORSTAMPS SHOROC, etc.) either operational or under 
development of being implemented. The DOT models must be 
viewed as a capability/application consistent with the 
total Training Command MIS and not as a separate system... 

ETE 

Page 26 ...The ETE is a general purpose simulation model and 
is thus not structured around a specific type of course. Its 
data base would not normally even approach the complexity 
of that of the TPF and SCRR. The data for a typical model 
run can be entered by the user at the start of a model 
session (and then stored for later recall or modification). 
Theoretically, the ETE could at present be applied to any 
ILS course or group of courses, regardless of location or 
even if the course is yet to be convened... 

...Factors favoring the development of this model included 
the following: (1) increasing numbers of ILS courses through¬ 
out EDTRACOM, (2) increasing resource constraints, (3) exis¬ 
tence of previous work in simulating ILS in the Navy, (4) 
prospects for a general purpose model that could immediately 
be put to use throughout EDTRACOM, (5) emphasis of other 
models on training process, as opposed to course design... 

Page 36 ...This problem area currently exists to a major 
degree in CNTT and will do so in the future at the TRACOMS. 
Present methods are seen as inadequate most of the time 
with substantial to great opportunity for improvement 
(million dollar area). Moderate to substantial changes in 
present organization/policies are seen as necessary to 
effect improvements... 

Page 7 ...and theoretically has applicability wherever ILS 
courses are designed, modified, or where a significant number 
of such courses must be managed given some commonality of 
resource requirements... 


III-16 










TAEG REPORT NO. 28 


Statement ■ 

■ Operational Feasibility (Cont'd) 


Page 51 ...While the ETE and highly conceptual ETAM models 
would appear to have the least impact external to the Command, 
they might have the greatest impact within. The magnitude 
of the individualization effort within CNTT alone, and the 
projected savings from this effort, would support continued 
RSD of these models... 

Page 42 .. .Although the data base of the ETE model might 
require frequent update, this should not present much of 
a management problem because the master data base would not 
be involved (only scratch disc). The ETE model per se would 
probably not be frequently updated... 

Page 8 ...The data required for the ETE are fairly simple in 
comparison to those needed by the other two models and their 
supply seems to present no operational problem... 

Page 26 .. .All of the major ILS decision variables can be 
represented and considerable user flexibility is provided. 

The ETE is generally formulated so that the user can formulate 
a specific model to represent a given ILS. The estimated 
time for such formulation (3-5 days) is acceptable... 

General 

Page 49 ...Specific level(s) and degree of application of 
the DOTS models cannot be defined at this time. The considera 
tions which preclude such definition are essentially the lack 
of costing information and lack of experience based on applica 
tion beyond FTC, Norfolk... 

Page 52 ...The group's consensus was that no additional per- 
sonnel would be needed. That while some training could be 
necessary, present staff can learn to use the models/results.. 

...This variable (additional personnel) is largely unknown at 
this point. It is, however, directly related to use which is 
a function of the acceptance and utility of the models them¬ 
selves. .. 

...Many existing hardware configurations now in place at 
potential user sites could be used for DOTS. The minimum 
requirements (i.e.,keyboards/printers) would suffice in many 
cases. CRTS and other more sophisticated peripherals would 
not be necessary in many locations. In a good many cases, 
communicating magnetic card typewriters could fulfill all 
the hardware requirements for access to the DOTS system, 
without significant impact on present use of this equipment. 
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Statement - Operational Feasibility (Cont'd) 


Commentary: 

The above quotes support the position that the three models are operationally 
feasible, considering the current NAVEDTRACOM organization, personnel, resources 
and decision-planning practices. This support is qualified by a T&E concern for 
the lack of an operational verification during Phase III. In short, under opera¬ 
tional conditions, do the models predict or simulate actual events accurately 
and, equally important, do they have significant utility to the appropriate users? 
This concern will be addressed under Criticism and Recommendations. 

The above position represents a T&E majority opinion, but it should be noted 
that the Naval Technical Training Command took serious exception to it. Their 
position will be stated in the CRITICISM AND RECOMMENDATIONS sub-section. 
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• Financial Feasibility (Part 4 - T&E) 

Statement - Financial Feasibility 

The three DOTS' models were not proven (nor disproven) to be finan¬ 
cially feasible, based on either current operational tasks displaced 
or costs avoided through model application. 

Based on empirical observation, costs of implementation appear 
to be minimal and, therefore, could probably be justified on the 
basis of probable benefits. 

Quotes: (T&E Final Report) 


Page 12 .. .As for proving the financial feasibility of applying 
modeling techniques in general throughout the EDTRACOM, it is 
not believed that this could presently be established ... 

Page 11 .. .Numerous attempts were made during the T&E to trace 
back from the application of the models to real problems 
emanating from the FTC and, in turn, to assess the benefits 
of supporting the solution of those problems. None of these 
attempts resulted in very useful information... 

Page 87 ...On the benefit side, perhaps the most meaningful 
indicator revealed in the T&E was provided by comments from 
the User T&E team that the modeling and MIS capabilities 
demonstrated by DOTS would be worth at least two billets at 
the training center level of implementation. As the technical 
feasibility section of this report pointed out, a field test 
is necessary to precisely identify a measure of benefit. 
However, the costs as estimated above are low enough to make 
application of the models a low-risk proposition, especially 
at the TRACOM level... 

...The greater question of financial feasibility concerns to 
what extent the results of DOTS can be extrapolated to assess 
the feasibility of applying such management techniques through¬ 
out the EDTRACOM in general. Since the techniques employed 
by the DOTS models (e.g., UP, simulation) are a fair sample of 
the field from which they are drawn, it seems reasonable to 
generalize the order of operating costs estimated above... 

Page 12 . . .Applications can differ so markedly with respect 
to cost and payoffs that any specific application hardly 
provides a basis for extrapolation of feasibility for the 
general case. The history of management science is strewn 
with examples of very good general models which never were 
implemented. Situational factors for each case seem to 
override any generalizations that might be made... 

— - 3 e — "'With the dearth of information on the benefit 
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Statement - Financial Feasibility (Cont'd) 


side of the feasibility question, it is best to focus on 
cost factors. Here it can be seen that the future cost 
of developing and operating the models may prove to be 
effectively less than even assessed by the contractor in 
his analysis. This position can be supported because of 
two concurrent developments to DOTS. First, the data 
base necessary for future model development is, in the main, 
being compiled regardless of DOTS. Thus, the cost of that 
data acquisition and maintenance need not be attributed to 
the DOTS models. Secondly, the TRACOMS either have or are 
in the process of procuring the necessary hardware to sup¬ 
port DOTS type models, again regardless of DOTS. For 
example, TRAPAC already has in operation a terminal with 
full telecommunications capability from which the models 
have already been accessed. Thus, it is doubtful that users 
would need to purchase equipment that they would not have 
procured otherwise... 

Page 49 ...Throughout the TSE we have been hampered by a 
lack of costing data. We are unable to predict, with any 
degree of validity, what this capability might cost in any 
of its terms (hardware, software, personnel, telecommunica¬ 
tions, site prep.) much less as a whole. This is not a 
criticism in that derivation of systems costs was not a goal 
of the effort to this point, but rather a statement of fact. 
This fact does serve to highlight the other issue, however, 
in that while activity level usage might be great in terms of 
numbers of interactions, does the utility warrant the invest¬ 
ment? We cannot say at this time. The most we can say is 
that if continued evaluation is directed, that a portion 
of the evaluation must address the economics of the system 
and that the economic evaluation must address alternative 
methods of application as well as levels... 

Page 53 .. . Given finding in first (hardware configuration) 
and fourth (number of ultimate weeks) discussion areas , what 
might be a. " ballpark" cost per incidence of. usage ratio . 

$20K a year estimate by IBM seems excessive if hardware is 
largely in place... 

Page 86 ...As a result of discussions with members of the 
user TSE team and visits to various EDTRACOM sites, it can 
be validly assumed that utilization of computer-based models 
like those of the DOTS project will not require special data 
or equipment acquisitions of a significant nature. From 
this it can also be assumed that the basic skills needed to 
operate the models will also be present at most sites. Since 
model operation and data maintenance which does not replace 
existing procedures is estimated to consume a very small 
number of manhours, it is not anticipated that support 
personnel cost will be appreciable. There will be a train¬ 
ing cost. On the basis of training conducted prior to the 


III-20 



























TAE6 REPORT NO. 28 


Statement - Financial Feasibility (Cont'd) 


on-site user TSE, this is estimated at about four man-weeks 
per site (2-4K). Data pertaining to time-sharing cost for 
the models were obtained from actual model operation at FTC, 
Norfolk (see TAEG Report 30 May 1975, User's Test Guide, 

Section V). Using these cost data and a somewhat arbitrary 
estimate of the extent of usage, an estimate of $10K per 
year was obtained (using all three models). With some 
amortization of training cost, this results in a total 
yearly cost that can be safely estimated at under $15K, 
allowing for some equipment depreciation (or rental cost 
allocation) and model maintenance... 

Page 49 .. .Discussions regarding systems costs and economy 
of operation centered, in the main, on whether or not school- 
house level implementation could ever be shown to be econom¬ 
ically feasible or even justifiable. This argument centered 
on the premise that the unavailability of precise costing data 
notwithstanding, the thrust of implementation should be toward 
the functional level in that only at this level would the 
magnitude and complexity of problems warrant the capability 
to apply modeling techniques... 

...Counter-arguments stated the premise that costs would, in 
all likelihood, be minimal to the point that they would not 
be the primary deterrent to systems implementation; and, further, 
that there were many ways in which the models could (should) 
be used at the activity level. That the amount of usage and 
attendant benefits to be derived from such use is limited only 
by the imagination and resourcefulness of the activity itself... 


Commentary: 

The IBM technical team agrees with the T&E Report: 
Quote (T&E Final Report): 


Page 11 ...It appears that a reasonable benefit/cost analysis 
of the models can only be derived from a valid field test. It 
must be established to what extent the models supplant existing 
procedures or personnel and what additional capabilities are 
provided - even more importantly what these capabilities buy 
in the way of greater effectiveness... 


This conclusion supports one of the recommendations evolving from a hypothetical 
benefit/cost feasibility exercise, based on theoretical assumptions, conducted 
during Phase II. That recommendation was: 
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Statement - Financial Feasibility - Commentary (Cont'd) 


Quote (Phase II Final Report):^ 


A decision to implement the DOTS models at the eighteen user 
locations identified in the preceding Projected Operational 
Responsibility sub-section, should be based on a thorough 
cost versus savings and avoidance justification. That 
justification and decision should not be made prior to the 
completion of DOTS Phase III and an assessment of actual 
operating results at the Norfolk Fleet Training Center test 
site. 

The Phase II DOTS validation exercise did provide sufficient 
indicators for a preliminary projection of cost savings and 
avoidances. Necessarily, this analysis had to be based 
primarily on subjective opinion and assumptions, due to lack 
of objective operational data. 

It should be noted that the referenced exercise during Phase II was based on 
the assumption that NAVEDTRACOM management practices would be modified to 
achieve maximum benefit from mathematical models, whereas the T&E team was 
tasked with assessing the DOTS models' financial implications as applied 
in the current environment. However, the Phase II recommendation that the 
cost justification take place after Phase III is equally valid for either 
assumption. 

The T&E Final Report expressed serious concerns pertinent to the Phase II pro¬ 
jection of costs and avoidances hypothesized for an integrated NAVEDTRACOM 
modeling-management system. Since the T&E team specifically excluded the 
concept of institutional change from its evaluation, these concerns should 
not be considered as pertinent to this statement of financial feasibility. 
However, the T&E team provided strong support for the Phase II recommenda¬ 
tion that a definitive cost avoidance exercise take place after Phase III. 

The T&E team's concerns will be addressed in more detail under the CRITICISMS 
AND RECOMMENDATIONS sub-section to follow. 


1 ^Design of Training Systems, Phase II Final Report . TAEG Report No. 
Vol. I, Section IV, December 1974, Page IV-19. 
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• Criticisms and Recommendations (Parts 1-4, T&E) 

The T&E Final Report contained a very comprehensive technical 
critique of the three models. This sub-section will not 
attempt to duplicate or summarize that technical material but 
will emphasize those criticisms and recommendations pertinent to 
the general DOTS' conclusions and long-range implications. 

Elimination of the technical critique does not imply disagree¬ 
ment. The IBM technical team felt the technical critiques and 
recommendations were valid and should be considered in any 
future DOTS effort. 

The format used in this sub-section is similar to that of previous 
ones. A statement, based on a consolidation of comments distributed 
through the T&E Report, is presented, followed by the actual quotes 
and that by an IBM commentary. Each quote is coded as follows: 

0 - Observation 

C - Criticism 

R - Recommendation 

As previously indicated, consolidation of dispersed quotes to 
support any given position statement can introduce a certain 
degree of bias. However, as long as the reader is aware of this 
approach and relies on the T&E Report for a more detailed analysis, 
this summary should highlight the key T&E concerns and recommenda¬ 
tions. 
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Statement #1 - Criticisms and Recommendations 

While the DOTS' effort appears to be complementary to existing and planned 
NAVEDTRACOM applications of Automated Data Systems (ADS), there is a need for 
significantly increased management and coordination of the composite ADS by 
CNET. This increase must take place if maximum benefit is going to be derived 
from any data base or modeling effort. 


Quotes (T&E Final Report): 


Page 47 (0) 

...From specific and general comments by attendees to this 
TSE, a strong display of feeling was evident that the EDTRA 
Command is in dire need of management in the area of systems 
development and coordination. Specifically, there is the 
current problem of a lack of coordination of systems develop¬ 
ment and interfacting. There is a requirement for an overall 
information systems management/development function that is 
independent from the operational administration of the ADP 
facilities in the EDTRACOM. Some form of information systems 
study/development function has to exist that addresses the 
needs of all levels of the Command, not just CNET... 

Page 41 (R) 

...Update/modification requirements in the case of upper 
management may center around problems involved in integrating 
DOTS with other information systems. A serious effort should 
be undertaken to determine complementary functions among 
these information systems if redundancies are to be minimized 
and efficient integration and upper management usage are to 
be attained. This effort is not part of the current DOTS 
mission or tasking... 

Page 37 (0) 

...A recurrent theme of the TSE discussions centered on the 
NITRAS data system (i.e., its shortcomings). The complementary 
characteristics of the DOTS data base management system was 
recognized... 

Page 14 (R) 

...The results of the DOTS project, primarily the application 
of data base management technology, should be integrated more 
closely with the work of the NETISA organization in the develop¬ 
ment of the NITRAS data base... 

Page 50 (0) 

...The central point is that DOTS is not to be another system 
separate and distinct from those already in being. It must 
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not be competitive. It must be complementary to a total MIS 
established from the myriad of (unintegrated) systems that 
now exists. Similarly, DOTS (as a function of the total 
Training Command MIS) must interface with those systems ex¬ 
ternal to the Training Command, such as SHORSTAMPS, that will 
impact on the Command... 

Page 87 (0) 

.. .However, a plus factor for products such as the DOTS models 
is the current state of management information systems (MIS) 
within the EDTRACOM. These are currently emergent and embryon¬ 
ic. Since the models are very complementary to the MIS develop¬ 
ment process, a fairly consistent MIS benefit might be projected 
from one development site to another. This underlying benefit 
might be capable of covering most risks associated with the 
more sophisticated utilities of models... 

Page 50 (0) 

...The need for accurate, timely planning is well recognized 
and there are several systems now operational or under opera¬ 
tional development and implementation which purport to provide 
this capability (none appear totally satisfactory be it a 
function of scope of concept or of performance)... 


Commentary - Statement #1: 

The above quotes emphasize a major concern of NAVEDTRACOM planners at all levels. 
That concern has to do with the lack of timely and accurate data required to re¬ 
spond to both operational and planning requirements. A related complaint, fre¬ 
quently voiced, was that lower levels provided significant amounts of data to 
higher ones but were unable to benefit from that effort through subsequent re¬ 
trieval . 

Although specifically excluding an evaluation of the DOTS Phase I effort from its 
assessment, the T&E team conclusions did support one of that Phase's major recom¬ 
mendations. The recommendation can best be described through the following random 
quotes extracted from the Phase I Final Report: 

Quote (Phase I Final Report):^ 

Page VI-328 ...It is important that methods be available to 
track key attributes of the system. For example, report 
accuracy and timeliness were observed to be a problem in 
a number of instances; however, there did not appear to be 


^ Design of Training Systems, Phase I Final Report, TAEG Report No. 12-1, 
December 1973. 
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any pressing management concern since backup systems were 
available. In the ideal system for managing training 
resources, quality, reliability/availability, timeliness, 
and usefulness of data would not only be available to 
measure the system, but they would also be integral to 
objective setting and measurement of training managers. 

Also, before the system effectiveness can be evaluated 
in terms of its role as a part of the overall resource 
management system, these data must be available... 

Page VI-341 ...Closely related to the preceding items is 
the need for faster access to resource data. This capability 
is especially important for detecting and analyzing variance, 
costing training plans, responding to queries, etc... 

Page V1-340 ...Resource management reports need to be made 
more timely, more accurate, and more relevant to the command 
level at which they are used. Performance reporting (actual 
vs plan) should have increased emphasis in terms of automated 
and timely feedback to reduce the reliance on manual logs 
and analyses... 

Page VI-304 ...Several locations were noted to have systemat¬ 
ically defined their data needs, others were in the process of 
defining systems to make more effective use of the existing 
data bases and to reduce the amount of manual activity in 
the preparation of reports...In an idealized training system 
there should be greater standardization and integration of 
methods, procedures, and equipments used in the development 
and processing of management information... 

Page VI-302 ...Resource and technical data must be supplied 
to the various systems throughout the reporting period so 
that meaningful data can be reported back to the activity 
for control purposes...a substantial amount of manual activity 
was noted. This was apparently due to the timing and accuracy 
of the basic reporting documents. It would seem from the 
comments that there are excessive errors contained in the 
input data which promulgates throughout the system. Thus, 
there is a real question as to how good a job is done to 
verify the reasonableness and/or accuracy of input data... 
However, an important aspect of increasing the effectiveness 
of resource control is to have resource managers assured 
that their reports are highly accurate... 

Page VI-343 ...Paperwork. It appears that a high percentage 
of the manual activity devoted to preparing basic documents 
could be more fruitfully applied in the analysis and solution 
of resource management problems. This, of course, implies 
either the elimination or the automation of much of it. 

Some of both appears desirable... 
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Page VI-74 ...Another facet of an ideal system is consis¬ 
tency and uniformity of designating an instructional organiza¬ 
tion. The large difference in the size of training - centers, 
schools, detachments, units - presents a most confusing 
picture. The origin of such differences is understandable 
since the present more centralized training structure picked 
up a number of relatively disparate elements. However, 
there is a need for a realignment of current terminology 
and organizations... 

Page VI-102 ...The idealized system requires optimal planning. 
Aspects of "optimum" include: uniformity in planning across 
organizations, use of equal cost/usage and other factors, 
capability to include contingencies, and the capability to 
project into a future time period...An initial requirement is 
a set of common planning factors for use by all elements of 
the NETS. Such factors must be centrally developed, main¬ 
tained, and controlled. In addition, uniform procedures 
for the utilization of these factors must be developed and 
made mandatory for all organizations... 

Page V1-340 ...The number of ways in which resources are cate¬ 
gorized, i .e., by program element, function sub-portions, 
budget classification code, element of expense, etc., seems 
unduly complicated for the most effective application of the 
system at all levels... 

Page VI-104 ...Factors should, as applicable, be expressed 
in terms of ranges or sets with associated assumptions to 
permit the planner to develop alternatives or to revise plans... 

Page VI-341 ...Realistic costing means the availability of 
historical costs and planning factors as well as a system for 
modifying them to the needs of any new costing exercise... 

Page VI-340 ...A more well-defined accounting unit is needed 
for planning, tracking, and controlling training. Present 
units, such as average under instruction, average on-board, 
and student man-months, are subject to varying interpretations 
which preclude an accurate measurement of actual training... 

Page VI-340 ...Greater and more uniform automatic data 
processing capability is required to adequate service all 
command levels. The non-uniformity and the absence of suf¬ 
ficient processing capability, however, is especially notice¬ 
able at the lower command levels... 


These Phase I recommendations and observations were based on assessment of the 
NAVEDTRACOM in support of the overall DOTS' thrust which envisioned an inte¬ 
grated multi-level system of mathematical models operating in a modified 
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management environment. A Phase I task was to identify the modifications 
required to achieve maximum benefit from such an approach. Lack of data was 
one of the major institutional gaps identified. 

That same gap is proving a serious concern in the day-to-day planning process. 
The IBM recommendations in SECTION IV of this report will amplify and support 
the T&E Report in this area. 
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The three DOTS models were not verified during Phase III. 

Note : DOTS' documentation defines validation as proving that the models did 
work as designed and verification as proving they accuractely simulated or pre¬ 
dicted actual events. In the quotes to follow, verification should be substi¬ 
tuted for validation. The T&E did support the Phase II validation but not 
verification, as was originally planned for Phase III. 


Quotes (T&E Final Report): 


Page 21 (0 and C) 

...(SCRR) .. .The basic LP technology and the MPSX software are, 
of course, not in need of validation due to their standardized 
nature. The validation documented by the model developer (TAEG 
Report 12-2, Vol. II, pp 11-30-60) demonstrate that the model 
is functioning properly. The model’s ability to accurately 
represent the real world is not demonstrated... 

Page 25 (0 and C) 

...(TPF) .. .The validation procedures for the TPF as described 
by the model developer (TAEG Report 12-2, Vol. II, IV, pp 22-30) 
do not constitute a true validation. The model is simply 
exercised in a number of conditions and its deterministic 
processes are allowed to respond accordingly...However, there 
is no basis for judging how well the model simulates the real 
training process flow. All that can be said is that the factors 
utilized in the TPF appear to be reasonable since they are drawn 
from those which are commonly used by the training managers... 

IBM Note : The logic validation of Phase II is confused 
with the intent to verify in Phase III. The referenced 
report does contain a logic validation description which 
did prove the model operated as designed. The above 
quote is correct in indicating that no proof of fidelity 
in simulating actual conditions (verification) was provided. 

Page 28 (0) 

...These conditions did not rule out validity testing using 
historical data. Unfortunately, the historical data available 
would not have been complete enough to allow even this type of 
testing... 

IBM Note : "These conditions" refer to those conditions 
precluding a meaningful field test. 
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Page 25 (0) 

...a true validation would have entailed the use of historical 
conditions and the matching of model output to actual outcomes 
in the training environment resulting from those conditions. 
There appears to have been no attempt at such a procedure. 

(Of course, this would be more of a validation of the planning 
factors used by the model than of the mechanics of the model 
per se.)... 

Page 27 (0 and R) 

...In the absence of ILS at FTC, Norfolk, the ETE was validated 
on the basis of a proposed consolidated EW school at Corry 
Station... 

...the ETE remains for the most part in an unvalidated state. 

It may be that, as the model developer states, there are no 
NAVEDTRACOM ILS applications that are presently suitable for 
use in validation. If this is still true, the condition 
should not prevail for long. The ETE should be validated 
against a number of different actual ILS cases as soon as 
possible... 


Commentary - Statement #2: 


The T&E Report is correct; very 
The original DOTS plan projected 
It was to have taken place in a 
by the IBM technical team on the 
Cape Kennedy Facility. Data was 
planned for this task, and that 
during Phase II using real-world 


limited verification was performed in Phase III, 
a major verification effort during this Phase, 
laboratory environment, i.e., verification runs 
IBM System/360 Model 40GF located at their 
to have been historical. Twelve months were 
task was to "...verify the models developed 
data and operational situations."' 5 


Based on an assessment of Phase I results, a significant change in approach 
was indicated early in Phase II. There were three factors indicating need 
for change. One, there seemed to be little need to expend R&D resources in 
proving that mathematical models could be designed which could accurately simu¬ 
late the types of conditions reflected by Phase I's candidate model list. Two, 
only limited historical data was available in a form amenable to the proposed 
experimental verification effort. Any conclusions based on a verification using 
the available data would have been subject to so many qualifications that its 
value would have been very limited. Three, the most important questions had 
to do with operational feasibility. Would models have value when applied to 
the real world of planning and decision-making? If they had value, could they 
be applied in the general environment considering human factors? 


15 Pesign of Training Systems, Phase I Final Report , TAEG Report No. 12-1, 
December 1973, Page 1-2. 
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Based on these three considerations, the Phase III verification in a laboratory 
environment was changed to a feasibility assessment of the three test models 
implemented in an operational test bed. The number of tasks required to in¬ 
stall the models in a time-sharing system, develop the required supporting 
software, and to train the test bed staff precluded also undertaking any signifi 
cant verification effort. To have reasonable validity, such a task would have 
required most of the planned twelve months. 

In no way does the above contradict the T&E team's recommendation for some form 
of verification. Whether the final decision is to implement the three DOTS' 
test models as is, modified versions, or totally different models, a historical 
verification should take place in a field test environment. 

The next Statement addresses such a test for the DOTS models. 


Ill-31 







TAEG REPORT NO. 28 


Statement #3 - Criticisms and Recommendations 

Utility of the DOTS test models has not been proven. 

Note : Utility has to do with the frequency the models would be applied to 
current planning and operational needs by a user, and the ultimate value of 
that application. 

Quotes (T&E Final Report): 

Page 57 (C) 

...Throughout the T&E there had been a dearth of feedback as 
to FTC, Norfolk, reactions to the specific utility of the 
various model information products, though some general reactions 
had been provided by the FTC DOT and the TRALANT T&E representa¬ 
tive. It was determined that while the FTC application was not 
a field test per se, the level of model development and oppor¬ 
tunity for application was such that specific examples of model 
utility using actual (if historical) problems, along with FTC 
management reactions, should have been produced in the course 
of DOTS Phase III... 

Page 28 (0) 

...an issue for the T&E is whether or not a field test should 
have been conducted in Phase III. It is the judgment of this 
T&E that such a field test could not have been comprehensively 
performed in Phase III. 

...It would not have been possible to obtain the user commit¬ 
ment necessary for proper field testing without first being 
able to demonstrate that the models were in working order, 
that they could be made to operate in the training environment, 
and that they addressed substantive training issues in a use¬ 
ful and meaningful manner... 

Page 55 (R) 

...The next phase of DOTS should include field test(s) of 
the developed models, under operational conditions... 

Page 12 (R) 

...Initiation of data collection and review for TRALANT/ 

TRAPAC application and field test of SCRR-TPF. This 
should include a clear assessment of opportunities for 
model contributions at these commands... 

Page 13 (R) 

...Presentation of the ETE model to the core of ILS course 
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developers in the EDTRACOM. This to be followed by 
field testing of the model in several cases of actual 
course design. An assessment of the benefits of the model 
should be made following these tests and the model should 
be further developed, revised, or put on the shelf for 
operational use as appropriate... 

Page 28 (R) 

...This field testing should, at a minimum, fill in 
some of the blanks left from this THE evaluation regarding 
more precise estimates of cost, accuracy, and utility. A 
suitable field test site for the ETE model is not so 
evident but must be pursued. It would not be wise to 
put that model on the shelf and wait for 1980 to happen... 


Commentary - Statement #3 

The IBM technical team agrees that utility has not been proven and should be 
demonstrated through additional field testing. 
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Maximum utility of mathematical models can only be achieved through applica¬ 
tion to higher command levels than that represented by the Norfolk Fleet 
Training Center. 

Note : The issue of level was a major one during the T&E with CNTECHTRA taking 
the strongest position. 

Quotes (T&E Report) 

Page 8 (0 and C) 

...The essential pitfall of the models as viewed by 
CNTT is that they "have been primarily designed to help that 
level of management least requiring assistance in decision¬ 
making" (i.e., the schoolhouse). CNTT construed this view 
as being supported by the acting Director of Training at 
FTC, Norfolk, who reported to the T&E team that the need 
to answer "what if" questions was rare (i.e., 5 or 6 times 
a year) at the FTC level. (Of course the Director also had 
many good things to say about the models. He recommended 
their continued development at the TRACOM level and hoped 
that the FTC could have continued access in the models on 
a periodic basis.)... 

...Comments on User T&E Results: While CNTT's point is 
sound in its premise (i.e., that the FTC site is atypical of 
the greater EDTRACOM), it has deviated somewhat in its con¬ 
clusion (i.e., that the operational feasibility of modeling 
l&s not been demonstrated)... 

Page 42 (R) 

...A functional flow chart should also be developed for the 
TRACOM showing scope and frequency of decision-making 
information/problem solving required at all levels, including 
specific examples of problems and methods/flow used to resolve 
these problems... 

Page 54 (R) 

...A general agreement evolved during discussion of user re¬ 
quirements to the effect that expansion of the prototypical 
modeling system to TRAPAC and CNTT activities is justified. 

Such expansion would address a wider variety and greater fre¬ 
quency of training decision-making conditions, explore the 
interfaces necessary to complement existing and developing 
systems, expand the data base, increase familiarity with 
modeling as a technique, potentially improve collection of 
related data, and provide better indications of system poten¬ 
tial than would an extensive effort to go operational at this 
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Page 55 (R) 

...The thrust of DOTS should be modified to include the 
investigation of applications at higher levels (i.e., beyond 
FTC) within CNET... 


Commentary - Statement #4 

Statement #4 is compatible with the recommendations of DOTS Phases I and II. 
The test model selection process, based on the Phase I study, illustrates this 
compatibility. The twenty-one potential test models were rated against nine 
criteria, one of which was: 


Quote (Phase I Final Report):’ 6 


Page VII-13-14 ...The organizational level, i.e., CNET, func¬ 
tional , or activity, which will benefit from the development 
of each candidate model was taken Into consideration in rating 
this criterion. In some cases, a single model or simple 
modifications of that model may benefit several levels of 
training management. A high correlation should exist between 
organizational level and savings potential, in that decisions 
made at higher management levels will generally impact a 
larger portion of the total training budget or of the total 
training population. Thus, models designed to assist in 
management decisions at several organizational levels received 
the highest rating. Models to be implemented at the CNET level 
received the next highest rating, while models to be implemented 
at the lower echelon level received the lowest rating... 


The DOTS Phase II report recommended an evolutionary implementation of mathemati¬ 
cal models to encompass three general levels of organizational and/or modeling 
complexity. 


Quotes (Phase II Report):^ 


Page IV-7 ...The long-range thrust of Technical Development 
Plan W^03 (P01A) is towards the integration of computerized 
mathematical models into all appropriate areas of the CNET's 


16 Pesign of Training Systems, Phase I Final Report , TAEG Report No. 12-1, 
December 1973, Pages VII-13-14. 

^Design of Training Systems, Phase II Final Report, TAEG Report No. 12-2, 
VolTiT^ecember 1574, Page '111-48.-- 
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decision analysis process. This thrust implies change in 
the way decisions are currently being made if the total 
decision process is to derive maximum benefit from the 
models. Therefore, an orderly and logical implementation 
of both the models and changes to the decision process itself 
are essential if the ultimate results are to be accepted 
and effectively used... 

...For purposes of estimating the resources required for the 
design, development, validation, and implementation of a total 
CNET's decision analysis system, models were arbitrarily 
divided into three levels. The hierarchy was based more on 
the functional use of the models than on the level of training 
command using the results of the models, although the two do 
tend to equate... 

...The SCRR, ETE, and TPF models are examples of the first 
level. They are primarily concerned with the projection of 
training resources and student flow. The development of a 
multi-level DOTS modeling system was initiated at the first 
level, since the horizontal implementation of these models 
across the Naval Education and Training System will require 
standardization of a basic data base and associated procedures. 
Such a standard will be essential to the support of higher 
level models. This same advantage will apply to the evolution 
from the second to the third level... 


The above quotes should only be interpreted as supporting the premise of 
Statement #4. The T&E Report took strong exception to the Phase II hypoth¬ 
esis that significant benefit could be achieved from an integrated multi¬ 
level modeling system operating within a modified management environment. 


Quotes (T&E Final Report): 


Page 14 (0) 

...It is a conviction here that, at least for the present, 
the successful application of management science technology 
is much more likely through many small scale applications... 


Page 9_ (0 and C) 

...In fairness to CNTT, it should be said that their 
alarm that someone might try to spread these particular models 
across the waters of EDTRACOM is not without cause. Much of 
the DOTS documentation concerning "projected operations" of 
the models invites this kind of fear (e.g., TAEG Report No. 12-2, 
Vol. I, Phase II Overview, pp IV-15 through IV-19). Such pre- 
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sentations seem to imply that the three models, with a minimum 
of modification, could be applied in a very systematic manner 
through much of CNET . This sort of thinking was further embel¬ 
lished by verbal presentations from IBM which emphasized that 
there would have to be fairly significant changes in the present 
Navy training organization in order to take full advantage of 
the models... 


The T&E's concern for Phase II's long-range recommendations and hypotheses 
will be addressed in the Commentary for Statement #5. The CNTECHTRA concern 
for the selection of Norfolk as a test bed is understandable since the T&E 
approached the models primarily as operational tools, as opposed to R&D test 
vehicles. This is not a criticism of the T&E team since their operational 
orientation and approach significantly increased the validity of the DOTS 
conclusions and recommendations as they apply to the current environment. 

In any event, the CNTECHTRA concern for the selection of Norfolk as a test 
bed merits an explanation. 

Two general principles guided definition of the final test environment. These 
were: 

1. The environment must permit a reasonable assessment of the DOTS' 
hypothesis. Although only a limited subset, its evaluation must 
permit extrapolation of conclusions to other sets or subsets of 
the system. 

2. The test environment and models selected, in addition to supporting 
hypothesis assessment, must provide potential for immediate 
practical return on the R&D investment. Test models, whether 
functional or mathematical, were to be selected considering their 
operational Implications if this consideration would not be detri¬ 
mental to achievement of DOTS' R&D objectives. 

The Norfolk FLETRACEN was selected during Phase II as an operational test bed 
for the following reasons: 

- The FLETRACEN management was agreeable to providing support for 
the test effort. 

- Although small compared to other possible test locations, the 
management activities and functions resident at the Norfolk 
FLETRACEN represented all of the test areas to be explored. 

- Due to its size, the additional data elements, beyond those 
provided by NITRAS and other sources, could be developed within 
the scope of the DOTS' effort. 

It must be emphasized that the priority DOTS' R&D objective was to assess 
the potential benefit, if any, to be derived from the application of manage- 
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ment science, operations research and mathematical modeling techniques to 
the total NAVEDTRACOM planning and decision process. To achieve this, a 
limited subset was to be modeled and the results evaluated. Extrapolations 
were to be drawn from this evaluation. Recommendations were to be developed 
which would complement the Strategic Assumptions evolving from Phase I which 
projected the NAVEDTRACOM environment through the mid-1980's. 

It became obvious during Phase I that test models could be selected that would 
serve as R&D test vehicles as well as providing more immediate value to existing 
operational management. However, the three test models were primarily selected 
based on their potential for evaluating the original DOTS' hypothesis and not for 
their immediate value to the Norfolk FLETRACEN. To partially compensate 
Norfolk for its interest and effort in supporting the test and evaluation, 
some limited software was developed to permit access to the DOTS' supplemental 
data base required to drive the test models. 
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The Phase II benefit/cost analysis is not valid. 

Note : No benefit/cost analysis, appropriate to the application of the three 
test models in today's management environment, was conducted by IBM. The T&E 
Report refers to a hypothetical projection of an integrated management/modeling 
system based on Phase I and II's strategic analysis and recommendations. 


Quotes (T&E Final Report): 


Page 11 (C) 

...A benefit/cost analysis was attempted in TAEG 
Report No. 12-2. A shortcoming is its assumption that the 
models as presently configured could be widely applied 
throughout CNET. As pointed out above, this premise must 
be rejected. There are other reasons why the benefit/cost 
analysis provided by the contractor is not acceptable. 

These will be identified in the detailed T&E Report... 

Page 86 (C) 

...The results of Phases II and III in no way warrant 
the usage assumptions made in that analysis. As a conse¬ 
quence, the resulting cost benefit conclusions are not viewed 
as acceptable... 

Page 54 (R) 

...If this system progresses to a recommendation for 
operational implementation, system specifics should be left 
to further determination during the procurement process and 
be done in such a way as to ensure investigation of all possible 
configurations with attendant costs... 

Commentary - Statement #5 

Notes from Pages 11 and 89 of the T&E Final Report assume that a definitive 
cost justification was attempted during Phase II. IBM did not attempt such 
a justification and recommended, at the end of Phase II, that one not be 
attempted until after Phase III. The following quotes from the Phase II 
exercise should put it in perspective: 


Quotes (Phase II Final Report):^ 

Page IV-7 ...The long-range thrust of Technical Development 


^ Design of Training Systems, Phase II Final Report , TAEG Report No. 12-2, 
Vol. 1, December 1974. 
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Plan P43-03(P01A) is towards the integration of computerized 
mathematical models into all appropriate areas of the CNET's 
decision analysis process. This thrust Implies change In 
the way decisions are currently being made If the total 
decision process is to derive maximum benefit from the 
models... 

Page IV-9 ...a forward pricing analysis was performed in 
Phase II, projecting the costs of a horizontal extension 
of the SCRR, ETE, and TPF models to form an integrated 
CNET modeling system... 

...Due to the number of variables and lack of operational 
experience, this pricing exercise must be considered highly 
speculative. DOTS' Phase III will provide experience in an 
operational environment, enabling a more precise definition. 

However, the projected costs presented here should be suffi¬ 
ciently accurate to support tactical planning... 

Page IV-19 ...A decision to implement the DOTS' models at the 
eighteen user locations identified in the preceding Projected 
Operational Responsibility sub-section should be based on a 
thorough cost versus savings and avoidance justification. 

That justification and decision should not be made prior 
to the completion of DOTS' Phase III and an assessment of 
actual operating results at the Norfolk Fleet Training 
Center test site... 

Page IV-21 ...As previously stated, this exercise should be 
repeated after more objective data have been developed during 
Phase III predictive validation of the three models... 

Page IV-16 ...In developing recommendations, it was assumed that 
the decision command levels Identified currently have, or will 
be granted, the authority to make the training resource decisions 
supported by the models... 

The Phase II analysis was not based on the assumption that the test models would 
be implemented without design change or, what is more important, change to the 
process by which the NAVEDTRACOM manages resources. 

In addition to developing and validating (did the models work as designed) the 
three test models. Phase II was tasked with developing the hypothetical impact 
of implementing a multi-level NAVEDTRACOM integrated modeling system concurrent 
with initiating change in the total command-management process to achieve maxi¬ 
mum benefit. This task was predicated on a Phase I Strategic Working Assump¬ 
tion (SWA). 
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Quote (Phase I Final Report):^ 


"A confluence of factors will result in a significant reduc¬ 
tion in the resources available to accomplish the NETS* train¬ 
ing mission. Although a current concern, the degree and scope 
of resource restriction will become increasingly severe through 
the mid-1980's and will result in major changes to the NETS* 
resource control approach and structure." 

"Economic necessity will dictate a reduction in the total 
resources required to accomplish the training mission as well 
as a reduction in the unit cost of training for any given task. 
All command levels of NETS* will experience increasing levels 
of inquiry and control from higher government sources. The 
following change statements are pertinent to this reduced 
resource availability: 

1. There will be an increased capability for cost effective¬ 
ness quantification, measurement and control. 

2. More systematic and comprehensive cost justification 
methods, techniques, programs, etc., will be developed 
and implemented. 

3. There will be an improved capability for marginal cost 
identification. 

4. Increased application of computer technology will result 
in a need for more uniform and consistent data bases." 

♦Note: "NETS," Naval Education and Training System, is now 
currently designated as NAVEDTRACOM.. 


This SWA was based on eighteen Strategic Assumptions ^ 0 also developed and 
analyzed during Phase I. Phase II's analysis was an attempt to make this 
general hypothesis more specific and amenable to empircal analysis. The key 
to the Phase II analysis is the Phase I assumption that severe resource 
restrictions "will result in major changes to the NETS resource control 
apprach and structure." 


^ Design of Training Systems, Phase I Final Report , TAEG Report No. 12-1, 

Vol. 1, December 1973, Pages IV-17 and 18. 

^ Design of Training Systems, Phase I Final Report , TAEG Report No. 12-1, Vol. 2, 
December 1973, Pages A 11-90 and 91. 


II1-41 







TAEG REPORT NO. 28 


Statement #5 - Criticisms and Recommendations - Commentary (Cont'd) 


The change projected by the SWA was formulated as a recommendation during 
Phase II. This was stated as: 21 

"Concurrent with model implementation, make the changes to 
the management system required to permit effective Implementa¬ 
tion of resource decisions. This implies granting various 
levels of training officials more authority over the use 
of training resources than they now have." 

The Phase II change assumption envisioned increased involvement of all 
NAVEDTRACOM commanders in the inspection and challenge of resource plans 
and decisions. Mathematical models were considered as both management tools 
to support this change and as change agents for facilitating its incorporation. 

To assist in an initial cost speculation, twenty potential users were identified 
at eighteen geographic locations 22 within the NAVEDTRACOM. Costs were extra¬ 
polated from Phase II experience with the three test models. 

The projected savings were based primarily on the change in management, 
philosophy and not on the application of the three test models in today's 
environment. Although it was assumed that an integrated modeling system 
would require a minimum of eighteen users, this was not the driving cost 
avoidance consideration. The driving hypothesis was: 

By increasing the degree of involvement by all training command 
levels in the resource planning and decision process, and by sup¬ 
porting that involvement with an integrated system of mathematical 
modeling tools, up to 2.1% of annual specialized training costs 
(student and NAVEDTRACOM) can be avoided. 

This 2.1% potential is projected as the "best case" condition, and even that 
is not projected until the fifth year after the management change and imple¬ 
mentation of an integrated modeling system. Worst case projects .2% the 
first year and 1.5% in the fifth year, as compared to a best case of .3% 
the first and 2.1% the second. 

Based on the more significant savings achieved in industrial training through 
application of similar techniques, the Phase II analysis should be considered 
extremely conservative. 

During the T&E, the team expressed a serious concern that the Phase II hypo¬ 
thetical analysis would be used to justify implementation of an integrated 
modeling system across the NAVEDTRACOM prior to a valid justification. Their 
concern is a reaonable one considering the number of systems and educational 
innovations that have been sold on the basis of just such hypotheses. This 
same concern was considered in documenting Phase II. The speculative nature 


Zl Qesign of Training Systems, Phase II Final Report , TAEG Report No. 12-2, 
Vol. 1, December 1974, Page V-1. 

22pes iqn of Training Systems, Phase II Final Report , TAEG Report No. 12-2, 
Vol. 1, December 1974, Page iV-l^. 
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Statement #5 - Criticisms and Recommendations - Commentary (Cont'd) 

of the analysis was repeatedly stressed, as well as the strong recommendation 
that a definitive cost justification take place after Phase III. 


The T&E team also expressed some concern for the feasibility of a change in 
management practice. This will be discussed in SECTION IV, Conclusions and 
Recommendations. 
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Statement #6 - Criticisms and Recommendations 

There is a need for establishment of a NAVEDTRACOM-wide program, under direction 
of a CNET agent, devoted to the development and coordination of management 
science techniques. 


Quotes (T&E Final Report) 


Page 13 (R) 

...At the conclusion of the TRAM application/field 
test, a program should be established devoted to the 
use and development of management science techniques to 
support training management throughout CNET. The agent 
for this program should reside within the CNET community 
but have close ties with the R&D sector. This group would 
serve as a two-way funnel for both translating CNET RSD 
requirements and for implementing advancements emanating 
from RSD. More importantly, the group would have the 
capability to directly address management support require¬ 
ments as possible within the state-of-the-art. This group 
could provide an invaluable element of continuity and con¬ 
sistency in the solution of such problems within CNET. 

The DOTS Phase I products, as corrected or amended, should 
provide an excellent foundation for this program. The 
models developed thus far would be the start of a "tool kit" 
of models or other techniques that could be applied to 
problems as they arise within the training community... 

Page 47 (0) 

...Additionally, the strong possibility exists that by and 
large the systems now under development or proposed are 
essentially designed to serve only the CNET level of need 
and are of little or no utility to the operational levels 
and functional levels of the EDTRACOM, except in several 
unique cases. Very little effort appears to be directed 
towards the development systems like DOTS which have the 
inherent capacity to become viable planning tools 
(i.e., for forecasting, modeling, etc.) or as tools 
for evaluating various approaches to resource utiliza¬ 
tion ... 


Commentary - Statement #6 

The Phase I recommendations provide strong support for Statement #6. Although 
intended to apply to all NAVEDTRACOM R&D requirements, the following summary 
quote is pertinent: 

"The approach to training research requires integration. 

The current fragmentary approaches do not yield the best results 
for the expenditure of funds. Central control under the leading 
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Statement #6 - Criticisms and Recommendations - Commentary (Cont'd) 


element in naval training (CNET) is essential. This must include 
systematic definition of problems, assignment of priorities, and 
prior commitment to implement effective results." 

Statement #6 is a sound recommendation. Additional reinforcement is provided 
on Pages VI-362-364 of the Phase I Final Report. 
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Statement #7 - Criticisms and Recommendations 

The cost of developing the three test models appears excessive. 


Quotes (T&E Report):- 
Page 87 (0 and C) 

...The costs incurred to develop the DOTS models appear 
inordinate and probably not supportable at anything less 
than command level implementation. But those costs in¬ 
cluded many one-time items necessary to initiation of 
the project... 

Page 14 (0 and C) 

...As a final remark, it should be noted that while the 
objective of the DOTS ADO is viewed as being achieved, 
the magnitude of expenditures that were required should 
be a lesson in the pitfalls of setting up projects which 
are a priori tasked with pleasing everybody... 


Commentary - Statement #7 

If DOTS is viewed as simply providing data processing solutions to current prob 
lems, Statement #7 is correct. However, this was not the intent of the DOTS 
effort, and costs during all three phases reflect its R&D objectives. 

The T&E team's comment appeared to be based on the assumption that all DOTS' 
Phases II and III expenditures were devoted to the selection, development and 
validation of the test model. 

The actual test model development took place during Phase II along with numer¬ 
ous other tasks related to the DOTS R&D goals. Phase III was more concerned 
with data base development, data base software support, command and control 
center installation, and training course development. Only a small percentage 
of Phase III resources were devoted to test model development. 

Although a precise breakdown is not possible, actual model development consumed 
less than 40% of resources expended during Phases I and II. 


General Comments - Criticisms and Recommendations 

The statements in this sub-section reflect the very constructive and substantial 
critique performed by the T&E team. Once again the team's detailed technical 
critique was not included, but was of equal quality and even greater scope. 

The only discontinuity resulted from the very pragmatic operational orienta¬ 
tion of the T&E team as it addressed the more hypothetical areas of command 

and management philosophy reflected in the conclusions and recommendations of 
Phases I and II. 
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SECTION IV 
PHASE III PRODUCTS 


CONCLUSIONS AND RECOMMENDATIONS 


Based on the experience gained through Phase III and the conclusions of the 
T&E Team, the following general conclusions can be made: 

• The Phase III Test and Evaluation demonstrated that DOTS type 
models have significant potential for improving the NAVEDTRACOM 
decision and planning process. 

• The NAVEDTRACOM Functional Levels will gain the greatest benefit 
from application of DOTS type mathematical models. 

• DOTS type models can probably be implemented using existing person¬ 
nel and without an Increase in billets. 

• The cost of model application will probably be significantly offset 
by Improved planning decisions. (This was not proven during Phase 
III and should be verified through additional field testing.) 

• It Is probable that the three DOTS test models, modified as 
suggested In the T&E Report's technical recommendations, will have 
utility value. (However, utility was not proven during the T&E and 
should be accomplished through a field test.) 

• The three DOTS models probably accurately simulate or predict actual 
conditions since they are based on the same planning parameters used 
by current planners. (Phase III did not produce an experimental 
verification of model fidelity, and this should be considered in a 
more extended field test.) 

e Two negative factors will initially retard acceptance. One, training 
managers are not able to quickly obtain needed data required for 
meeting their basic planning and decision responsibilities, and this 
will have to be resolved prior to implementation of sophisticated 
models. Two, models could be improperly used by higher level manage¬ 
ment to justify arbitrary resource decisions. Since no model can 
include all of the judgmental factors considered by the lower level 
decision maker, this improper use could have serious implications. 

• Proper training of all impacted managerial and operational personnel 
is essential to achieving acceptance and use of mathematical models. 
Training must be supported by a guided period of practical use in 
addition to formal theory and laboratory sessions. 
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The following recommendations are divided into two major areas. The first will 
address possible operational enhancements to the DOTS test models, and also 
desirable tasks for enhancing operational use and acceptance. The second area 
will cover possible institutional changes that could lead to a significant in¬ 
crease in the benefits to be derived from mathematical modeling. 

• Operational Recommendations 

The following represents a summary of the operational and technical 

recommendations identified during the Test and Evaluation: 

1. Initiate data collection and review for TRALANT/TRAPAC applica¬ 
tion and field test of SCRR-TPF. This should Include a clear 
assessment of opportunities for model contributions at these 
commands. Each command should designate an individual from its 
staff who will be the primary interface point in model develop¬ 
ment and have a clear responsibility for seeing that the models 
get put to real and practical use with the command to the extent 
possible. 

2. Modify the SCRR model as necessary in order to apply It at 
the TRACOM level. 

3. Review and modify the TPF model in terms of the following con¬ 
siderations: 

- Capabilities of a DBMS (e.g., RAMIS) to supplant the TPF 

in its current functions (given that no satisfactory statis¬ 
tical parameters can be developed). 

- Possibilities for more efficient model design given that 
the week-by-week simulation is equivalent, in effect, to 
successive transformations of course parameters using vectors 
of proportionality factors on a quarterly basis. 

- The possibility for alternative approaches to the develop¬ 
ment of statistically based model parameters. 

- Modifications as necessary to accommodate TRACOM level 
application. 

4. Design and conduct field tests of the SCRR-TPF at TRALANT and 
TRAPAC. 

5. Present the ETE model to the core of ILS course developers in 
the EDTRACOM. This is to be followed by field testing of the 
model in several cases of actual course design. An assessment 
of the benefits of the model should be made following these 
tests and the model should be further developed, revised, or 
put on the shelf for operational use as appropriate. 

6. After verification of fidelity and utility, conduct a cost/benefit 
analysis. 
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7. If operational implementation is indicated by the field test, 
it should be preceded by an in-depth training program and a 
period of closely supervised practical application. 
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• Institutional Recommendations 


The Government T&E was primarily concerned with the application of 
the three test models In the current NAVEDTRACOM decision and planning 
process. In addition, IBM was tasked with extrapolating the use of 
mathematical models into a modified NAVEDTRACOM process. This Phase 
III Report will not duplicate the Phase I Functional Recommendations, 
but will expand and qualify those Phase II recommendations having to 
do with changes to the NAVEDTRACOM management process. 

Phases I and II identified the following assumptions about the way 
the NAVEDTRACOM manages training and training resources. 

- Generally, training plans and decisions are formulated by 
the technical staffs and approved by the appropriate command 
level. 

- As long as these plans and decisions fall within acceptable 
fiscal and other resource limits, and are responsive to user 
needs, there is seldom a challenge from the authorizing com¬ 
mander concerning alternative possibilities. 

- Under the conditions described above, for all practical 
purpose, there is never a challenge from higher command 
echelons forcing consideration of alternatives by the 
approving commander. 

- Challenges do take place at all command levels when there 
is a need to withdraw resources from a training organiza¬ 
tion or when an organization requests additional resources. 

- Generally, the catalysts precipitating the challenges 
described in the previous assumption are such stimuli as 
changes in allocations, training requirements, or other 
unanticipated system drivers. 

- There is a natural tendency, as in any organization, for 
any given level of command to limit precise visibility to 
higher levels of its resource utilization and inventory. 

The resultant of these assumptions is a lack of command pressure to 
achieve efficiencies through systematic challenges as long as the 
NAVEDTRACOM can meet training requirements within a given funding 
level. This also represents the major difference between the way 
training functions are managed in industry as compared to the 
military. In industry, whether supervisory or executive level, 
management is required to become very involved in resource as well 
as people management. Traditionally, the military services have 
separated command (getting things done through people or tactical 
resources) and resource management (planning and control of costs 
and non-tactical resources). Most military commanders at any level, 
in fact, avoid those tasks associated with resource management or 
"bean counting" based on the assumption that to do so would reduce 
their effectiveness in achieving primary missions. 
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It cannot be automatically assumed that application of those manage¬ 
ment techniques used by industry to reduce training costs should be 
applied to the NAVEDTRACOM. Also, the assumptions previously stated 
should not be automatically construed as criticisms of the NAVEDTRACOM 
management staff. However, if the NAVEDTRACOM is to achieve the maxi¬ 
mum potential benefits in terms of cost effectiveness through applica¬ 
tion of mathematical modeling, all levels of command will be required 
to adopt their industrial counterpart's techniques of systematic 
challenge and analysis of resource plans and decisions. 

The following three recommendations are based on this assumption: 

1. Revise current management practices and motivational approaches 
to create an environment amenable to acceptance and use of the 
recommended management techniques. These revisions are envisioned 
as being reasonable ones, but will require changes in some sensi¬ 
tive areas. As examples: commanders at all levels will need to 
become more knowledgeable of the training planning process and 

its results; Fitness Reports will need to emphasize this shift 
in approach; commanders will have to become more comfortable with 
some of the more hypothetical exercises of managerial science 
and operations research; etc. 

2. Provide increased multi-level visibility of training resource 
utilization and inventory to all appropriate levels. Current 
projects within the NAVEDTRACOM, under general direction of the 
CNET Naval Education and Training Information Service Agency 
(NETISA), should eventually provide a basic informational 
capability. It must be emphasized that this capability is not 
only essential to successful Implementation of the DOTS' 
institutional recommendations, but also to those concerning 
the implementation and application of mathematical modeling 
tools in the current environment. 

3. Develop mathematical models and other management tools based 

on the data capability recommended under the second recommenda¬ 
tion, and designed to support the first recommendation's 
management practices. 

In developing these recommendations, the distinction between "command" 
and "management" was recognized. It is understood that if these recom¬ 
mendations are implemented, commanders will be required to practice 
certain managerial approaches they now leave to their technical staffs. 
It is also recognized that this shift may not be considered as one of 
"reasonable change" by the commands impacted. However, the recommended 
managerial practices applied without the direct force of command 
authority, inspection and control, will have limited value except as 
they assist the technical staffs to meet the procedural requirements 
of the Planning, Programming and Budgeting System. 

It should be stressed that the T&E Team, although not rejecting the 
DOTS' hypothesis of "reasonable change," did imply that It would 
require changes too significant to be considered reasonable. The 
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team's conclusion was not based on an assessment of the DOTS' Insti¬ 
tutional recommendations as a formal part of the T&E, but was a general 
observation. 

In support of the T&E Team's position, there is a growing concern 
within the military for the current trend towards equating command to 
management. This is best illustrated through quotes from an article^ 
by General Lucius D. Clay, USAF: 

"Management has a proper place in operation of a military 
service, but management must be recognized for what it is - 
a system of bookkeeping that is primarily associated with 
statistics." 

"Command is the relationship between people. People do 
things." 

"...let us not usurp the traditional functions of the commander 
under the banner of management." 

General Clay makes an excellent point. However, if the DOTS strategic 
assumptions pertinent to severe resource restrictions through the mid- 
1980's have validity, military command may be forced into the resource 
management role. If this happens, mathematical models and their associ¬ 
ated data bases may become essential tools for all command levels as 
well as for those staff levels concerned only with resource planning. 


23 Clay, Gen. Lucius D., USAF, Management Is Not Command, Air Force Maaa?inp. 
Sept. 9, 1975, Page 63. 
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PHASE III DOCUMENTATION 


In addition to the T&E and this Phase III Final Report, which together constitute 
documentation of Phase Ill's R&D outcomes, two additional manuals were produced 
during Phase III. 

This sub-section will provide a brief description of those documents, developed 
during DOTS Phase III, describing the three test models and providing sufficient 
operating procedures to permit exercising the models and revalidating their logic 
designs. The documents are entitled USER'S MANUAL and PROGRAM MAINTENANCE MANUAL. 
The Phase III documentation task was intended to provide sufficient technical 
information to permit future testing and possible model revision and development 
for inclusion in an operational environment. 

Some limited data base capabilities were developed to support the DOTS' test effort. 
These are also described. 

The DOTS test bed included installation of the models in a commercial time-sharing 
environment. To reduce the cost of establishing a simulated operational test en¬ 
vironment, as well as reducing the cost of actual model development, maximum use 
was made of existing program products. Documentation supporting these is not in¬ 
cluded in the Phase III documentation, but adequate references are provided as 
appropriate. 

A description of each of the two Phase III manuals follows. It should be noted 
that these manuals supersede those volumes of the DOTS Phase II Final Report 
covering model operations and programming description. 
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t PROGRAM MAINTENANCE MANUAL (TAEG REPORT NO. 29) 

The Program Maintenance Manual provides detailed information on 
the level of the three DOTS models and the DOTS data base at the 
end of Phase III. The level of information is sufficient to provide 
a base for programmers intending to install or modify the DOTS models. 

The manual is organized around the four major DOTS sections, the DOTS 
data base, the Educational Technology Evaluation (ETE) model, the 
System Capabilities Retirements and Resources (SCRR) model, and the 
Training Process Flow (TPF) model. The three models function inde¬ 
pendently and the data base (although it serves both the SCRR and TPF 
models) can also be considered as an independent sub-system. There¬ 
fore, each of the models and the data base are discussed In a separate 
section of the manual. 


Within each section, the sub-system being discussed is subdivided 
into its major components. Each section contains a sub-system de¬ 
scription and macro-level flow, detailed flows and program listings 
of the sub-system components. For FORTRAN and PL/1 programs, the 
detailed flow charts were generated using the Autoflow II* System. 

The manual is sub-divided as follows: 

I. INTRODUCTION 

A. Purpose 

B. Organization 

II. DOTS DATA BASE 

A. Introduction and Top Level Description 
1. Top Level Flows 

B. Data Base Sub-system A 

1. Sub-system Top Level Flows 

2. Sub-system Program Description 

3. Autoflow Diagrams, Executive Flows 

4. Program Executive Listings 

III. EDUCATIONAL TECHNOLOGY EVALUATION MODEL 


A. 

B. 

C. 

D. 

*Autoflow II is 


Sub-system Top Level Flows 
Sub-system Program Description 
Autoflow Diagrams, Executive Flows 
Program Executive Listings 
a product of Applied Research, Inc. 
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IV. SYSTEM CAPABILITIES/REQUIREMENTS AND RESOURCES MODEL 

(Same as Section III) 

V. TRAINING PROCESS FLOW MODEL 

(Same as Section III) 
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• USER'S MANUAL (TAEG REPORT NO. 30) 

The primary purpose of the User's Manual is to provide the DOTS model 
user with sufficient information to procedurally execute the models 
and take advantage of the supporting data base. In addition, test 
procedures are provided for each model to permit user retesting of 
functional validity. 

The manual also provides an overview of model architecture and logic. 
Descriptions of input parameters and outputs are included. 

The manual is subdivided as follows: 

I. INTRODUCTION 

A. Purpose 

B. ' Organization 

C. System Overview 

D. General System Procedures 

II. DOTS DATA BASE 

A. Architecture 

B. Maintenance 

1. Administrative Procedures 

2. Operational Procedures 

C. Inquiry Procedures 

III. EDUCATIONAL TECHNOLOGY EVALUATION MODEL 

A. Model Architecture 

1. Model Description 

2. Assumptions and Logic Design 

B. Input Parameter Description 

C. Output Parameter Description 

D. Operational Procedures 

E. Operational Test 

IV. SYSTEM CAPABILITIES/REQUIREMENTS AND RESOURCES MODEL 

(Same as Section III) 

V. TRAINING PROCESS FLOW MODEL 

(Same as Section III) 
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